Observability Metrics
Endpoints
-
https://api.eu.cloud.talend.com
URL for the AWS Europe region
-
https://api.ap.cloud.talend.com
URL for the AWS Asia Pacific region
-
https://api.us.cloud.talend.com
URL for the AWS United States East region
-
https://api.au.cloud.talend.com
URL for the AWS Australia region
-
https://api.us-west.cloud.talend.com
URL for the Azure United States West region
Security scheme
This scheme can be referenced across the API
Public
Name | Description |
---|
Get component metrics of task runs
GET /monitoring/observability/executions/{executionId}/componentThis endpoint is used to get component metrics of a given task run.
To retrieve your metrics data accurately, the process may vary in duration.
Security
Request
Name | Description | Type | Attributes and examples |
---|---|---|---|
executionId | Task Execution ID | string Required |
Name | Description | Type | Attributes and examples |
---|---|---|---|
offset | The start position of the pagination of the entries, greater or equal to 0 | integer Optional |
INT32 0 |
limit | Number of entries to return. The default value is 50 and the maximum value is 200 | integer Optional |
50 INT32 50 |
sortBy | The field by which the returned entries are sorted. The default value is component_start_time_seconds | string Optional |
component_start_time_seconds component_start_time_seconds |
sortOrder | The way in which the returned logs are sorted, desc or asc Whether the order is ascending or descending, a null or an empty field is always placed at the end of the list to be returned in the response |
string Optional |
asc asc |
Name | Description | Type | Attributes and examples |
---|---|---|---|
talend-version | string Optional |
2021-03 |
Response
400Status 400
{
"status": 400,
"message": "Task execution ID is not valid id.",
"requestId": "f1f6d388-b114-4d02-8d90-da2c8d1af60f"
}
401Status 401
{
"status": 401,
"message": "Unauthorized"
}
403Status 403
{
"status": 403,
"message": "Forbidden",
"requestId": "8c7c7de7-0a9a-45be-9e28-f0a34e6007fd"
}
404Status 404
{
"status": 404,
"message": "Task execution with ID '83503269-d6a1-437f-bf91-d38ad74a1ecf' not found.",
"requestId": "281f0387-9e88-4879-835d-30c1f3509af3"
}
500Status 500
{
"status": 500,
"message": "Internal Server Error",
"requestId": "2b6e98ae-66dd-4895-8f04-dc2f85617855"
}
Search raw component metrics
POST /monitoring/observability/metrics/componentUse this API endpoint to query raw component metrics. You can make up to 5 queries per minute.
To retrieve your metrics data accurately, the process duration may vary.
Request
Name | Description | Type | Attributes and examples |
---|---|---|---|
talend-version | string Optional |
2021-03 |
Response
400Status 400
{
"status": 400,
"message": "Task execution ID is not valid id.",
"requestId": "f1f6d388-b114-4d02-8d90-da2c8d1af60f"
}
401Status 401
{
"status": 401,
"message": "Unauthorized"
}
403Status 403
{
"status": 403,
"message": "Forbidden",
"requestId": "8c7c7de7-0a9a-45be-9e28-f0a34e6007fd"
}
500Status 500
{
"status": 500,
"message": "Internal Server Error",
"requestId": "2b6e98ae-66dd-4895-8f04-dc2f85617855"
}
ComponentQueryRequest
Name | Description | Type | Attributes and examples | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
environmentId | Environment ID | string Required |
610d3b7da8abfa32ffb6506 |
|||||||||
startTime | Start time of metrics generation and collection. This time corresponds approximately to when your Job task starts to work.
|
string Required |
2021-09-08T06:58:04.604Z |
|||||||||
endTime | End time of metrics generation and collection. This time corresponds approximately to when this component in your Job task finishes its work
|
string Required |
2021-09-08T06:59:04.804Z |
|||||||||
limit | Page size, 1~200 |
number Optional |
50 |
|||||||||
offset | Page offset, >=0 |
number Optional | ||||||||||
sortBy | Sort by field | string Optional |
component_start_time_seconds |
|||||||||
sortOrder | Sort order | string Optional |
asc |
|||||||||
filters | array of ComponentFieldFilter Optional |
|
{
"environmentId": "6093f6a36f01d639680d31cd",
"startTime": "2021-09-08T06:58:04.604Z",
"endTime": "2021-09-08T06:58:06.463Z",
"limit": 50,
"offset": 0,
"sortBy": "component_start_time_seconds",
"sortOrder": "asc",
"filters": [
{
"field": "engine_id",
"operator": "in",
"value": ["a60bb1c0-7669-407f-9326-138af05da18a"]
},
{
"field": "component_start_time_seconds",
"operator": "gte",
"value": 1628266578
}
]
}
ComponentFieldFilter
Name | Description | Type | Attributes and examples |
---|---|---|---|
field | Field filter name | string Required |
|
operator | Field filter operator
|
string Required |
|
value | Field filter value For operator
For operator
|
ObjectRequired | |
{
"field": "engine_id",
"operator": "in",
"value": ["a60bb1c0-7669-407f-9326-138af05da18a"]
}
ExecutionComponentMetrics
Name | Description | Type | Attributes and examples | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
engine_id | Engine ID | string Required |
a60bb1c0-7669-407f-9326-138af05da18a |
|||||||||
workspace_id | Workspace ID | string Required |
61273932d0366133d05729b7 |
|||||||||
task_id | Task ID | string Required |
612739e79a0ac71b8f3ed4dd |
|||||||||
task_execution_id | Task execution ID | string Required |
947e3e2f-d199-4988-a5ab-14ceb36c80f3 |
|||||||||
artifact_id | Artifact ID | string Required |
612739e79a0ac71b8f3ed4db |
|||||||||
artifact_name | Artifact name | string Optional |
job_with_rejected_rows |
|||||||||
artifact_version | Artifact version | string Required |
0.1.0.20212608065119 |
|||||||||
start_time | Jon run start time | string Required |
2021-08-26T06:53:30.127Z |
|||||||||
finish_time | Job run finish time | string Optional |
2021-08-26T06:53:35.361Z |
|||||||||
rows_rejected | Number of rows rejected | number Optional |
1 |
|||||||||
operator | Operator | string Optional |
admin |
|||||||||
operator_type | Operator type | string Optional |
HUMAN |
|||||||||
processes | array of ProcessMetric Required |
|
||||||||||
metrics | ComponentMetricPage Required |
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "61273932d0366133d05729b7",
"task_id": "612739e79a0ac71b8f3ed4dd",
"task_execution_id": "947e3e2f-d199-4988-a5ab-14ceb36c80f3",
"artifact_id": "612739e79a0ac71b8f3ed4db",
"artifact_name": "job_with_rejected_rows",
"artifact_version": "0.1.0.20212608065119",
"start_time": "2021-08-26T06:53:30.127Z",
"finish_time": "2021-08-26T06:53:35.361Z",
"rows_rejected": 1,
"operator": "admin",
"operator_type": "HUMAN",
"processes": [
{
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h"
}
],
"metrics": {
"items": [
{
"pid": "20210806181617_2Y68h",
"connector_type": "tMongoDBConnection",
"connector_label": "tMongoDBConnection_1",
"connector_id": "tMongoDBConnection_1",
"component_start_time_seconds": 1628266578
},
{
"pid": "20210806181617_2Y68h",
"connector_type": "tRowGenerator",
"connector_label": "tRowGenerator_1",
"connector_id": "tRowGenerator_1",
"target_connector_type": "tFlowMeter",
"target_label": "vFlowMeter_row1",
"target_id": "vFlowMeter_row1",
"component_start_time_seconds": 1628266578,
"component_connection_rows_total": 5000000,
"component_execution_duration_milliseconds": 491585
},
{
"pid": "20210806181617_2Y68h",
"connector_type": "tFlowMeter",
"connector_label": "vFlowMeter_row1",
"connector_id": "vFlowMeter_row1",
"component_start_time_seconds": 1628266578
},
{
"pid": "20210806181617_2Y68h",
"connector_type": "tFlowMeter",
"connector_label": "vFlowMeter_row1",
"connector_id": "vFlowMeter_row1",
"target_connector_type": "tMongoDBOutput",
"target_label": "Insert from SQL",
"target_id": "tMongoDBOutput_1",
"component_start_time_seconds": 1628266578,
"component_connection_rows_total": 5000000,
"component_execution_duration_milliseconds": 491605
},
{
"pid": "20210806181617_2Y68h",
"connector_type": "tMongoDBOutput",
"connector_label": "Insert from SQL",
"connector_id": "tMongoDBOutput_1",
"component_start_time_seconds": 1628266578
},
{
"pid": "20210806181617_2Y68h",
"connector_type": "tMongoDBInput",
"connector_label": "tMongoDBInput_2",
"connector_id": "tMongoDBInput_2",
"target_connector_type": "tLogRow",
"target_label": "tLogRow_2",
"target_id": "tLogRow_2",
"component_start_time_seconds": 1628267070,
"component_connection_rows_total": 2158754
},
{
"pid": "20210806181617_2Y68h",
"connector_type": "tLogRow",
"connector_label": "tLogRow_2",
"connector_id": "tLogRow_2",
"component_start_time_seconds": 1628267070
}
],
"limit": 50,
"offset": 0,
"total": 7
}
}
ProcessMetric
Name | Description | Type | Attributes and examples |
---|---|---|---|
process_id | Process ID. This is JVM’s UUID | string Required |
c151b20d-cb9a-34ad-b219-2080761dd193 |
job_name | Job Name | string Required |
sub_job_2 |
pid | PID. This is the unique Job identifier for the current run | string Required |
WTHlkK |
father_pid | ID of the first direct parent Job | string Required |
20210827095720_E61Q1 |
root_pid | ID of the first root Job if there are several Subjobs | string Required |
20210827095720_E61Q1 |
{
"process_id" : "c151b20d-cb9a-34ad-b219-2080761dd193",
"job_name" : "sub_job_2",
"pid" : "WTHlkK",
"father_pid" : "20210827095720_E61Q1",
"root_pid" : "20210827095720_E61Q1"
}
ComponentMetricPage
Name | Description | Type | Attributes and examples | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
items | array of ComponentMetric Required |
|
||||||||||
limit | Maximum number of entries to be returned on a page in the response | number Required |
50 |
|||||||||
offset | The start position of the pagination of the entries, greater or equal to 0 | number Required |
0 |
|||||||||
total | Total number of entries in the response | number Required |
500 |
{
"items": [
{
"pid": "20210827095720_E61Q1",
"connector_type": "tParallelize",
"connector_label": "tParallelize_1",
"connector_id": "tParallelize_1",
"component_start_time_seconds": 1630029441
},
{
"pid": "20210827095720_E61Q1",
"connector_type": "tJava",
"connector_label": "tJava_1",
"connector_id": "tJava_1",
"component_start_time_seconds": 1630029441
},
{
"pid": "20210827095720_E61Q1",
"connector_type": "tJava",
"connector_label": "tJava_2",
"connector_id": "tJava_2",
"component_start_time_seconds": 1630029441
},
{
"pid": "20210827095720_E61Q1",
"connector_type": "tRunJob",
"connector_label": "tRunJob_1",
"connector_id": "tRunJob_1",
"component_start_time_seconds": 1630029441
},
{
"pid": "20210827095720_E61Q1",
"connector_type": "tRunJob",
"connector_label": "tRunJob_2",
"connector_id": "tRunJob_2",
"component_start_time_seconds": 1630029441
},
{
"pid": "WTHlkK",
"connector_type": "tRunJob",
"connector_label": "tRunJob_1_subjob2",
"connector_id": "tRunJob_1",
"component_start_time_seconds": 1630029441
},
{
"pid": "W1yeMB",
"connector_type": "tParallelize",
"connector_label": "tParallelize_1_subjob1",
"connector_id": "tParallelize_1",
"component_start_time_seconds": 1630029441
},
{
"pid": "W1yeMB",
"connector_type": "tJava",
"connector_label": "tJava_1_subjob1",
"connector_id": "tJava_1",
"component_start_time_seconds": 1630029441
},
{
"pid": "W1yeMB",
"connector_type": "tJava",
"connector_label": "tJava_2_subjob1",
"connector_id": "tJava_2",
"component_start_time_seconds": 1630029441
},
{
"pid": "OPsysU",
"connector_type": "tStatCatcher",
"connector_label": "tStatCatcher_1",
"connector_id": "tStatCatcher_1",
"target_connector_type": "tLogRow",
"target_label": "tLogRow_2",
"target_id": "tLogRow_2",
"component_start_time_seconds": 1630029441,
"component_connection_rows_total": 1,
"component_execution_duration_milliseconds": 2
},
{
"pid": "OPsysU",
"connector_type": "tLogRow",
"connector_label": "tLogRow_2",
"connector_id": "tLogRow_2",
"component_start_time_seconds": 1630029441
}
],
"limit": 50,
"offset": 0,
"total": 100
}
ComponentMetric
Name | Description | Type | Attributes and examples |
---|---|---|---|
pid | Unique Job identifier for the current run | string Required |
OPsysU |
connector_type | Connector type | string Required |
tStatCatcher |
connector_label | Connector label | string Required |
tStatCatcher_1 |
connector_id | Connector ID | string Required |
tStatCatcher_1 |
target_connector_type | Target connector type | string Optional |
tLogRow |
target_label | Target connector label | string Optional |
tLogRow_2 |
target_id | Target connector ID | string Optional |
tLogRow_2 |
component_start_time_seconds | Time stamp indicating the component start time (any component in the Job) | number Required |
1628266578 |
component_connection_rows_total | Total number of rows treated by a component in the Job | number Optional |
5000000 |
component_execution_duration_milliseconds | Amount of time it takes to execute a Job component (generated for components that are managing rows) | number Optional |
491585 |
stacktrace | Contains the component stacktrace (only if an error occurs while executing the component) | string Optional |
NullPointer |
{
"pid" : "OPsysU",
"connector_type" : "tStatCatcher",
"connector_label" : "tStatCatcher_1",
"connector_id" : "tStatCatcher_1",
"target_connector_type" : "tLogRow",
"target_label" : "tLogRow_2",
"target_id" : "tLogRow_2",
"component_start_time_seconds" : 1630029441,
"component_connection_rows_total" : 5000000,
"component_execution_duration_milliseconds" : 491585,
"stacktrace" : "NullPointer"
}
RawComponentMetricPage
Name | Description | Type | Attributes and examples | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
items | array of RawComponentMetric Required |
|
||||||||||
limit | Page size | integer Required |
INT32 50 |
|||||||||
offset | Page offset | integer Required |
INT32 0 |
|||||||||
total | Total number | integer Required |
INT32 500 |
{
"items": [
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tMongoDBConnection",
"connector_label": "tMongoDBConnection_1",
"connector_id": "tMongoDBConnection_1",
"component_start_time_seconds": 1628266578
},
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tRowGenerator",
"connector_label": "tRowGenerator_1",
"connector_id": "tRowGenerator_1",
"target_connector_type": "tFlowMeter",
"target_label": "vFlowMeter_row1",
"target_id": "vFlowMeter_row1",
"connection_type": "row1",
"connection_name": "output",
"component_start_time_seconds": 1628266578,
"component_connection_rows_total": 5000000,
"component_execution_duration_milliseconds": 491585
},
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tFlowMeter",
"connector_label": "vFlowMeter_row1",
"connector_id": "vFlowMeter_row1",
"component_start_time_seconds": 1628266578
},
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tFlowMeter",
"connector_label": "vFlowMeter_row1",
"connector_id": "vFlowMeter_row1",
"target_connector_type": "tMongoDBOutput",
"target_label": "Insert from SQL",
"target_id": "tMongoDBOutput_1",
"connection_type": "meterRowrow1",
"connection_name": "output",
"component_start_time_seconds": 1628266578,
"component_connection_rows_total": 5000000,
"component_execution_duration_milliseconds": 491605
},
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tMongoDBOutput",
"connector_label": "Insert from SQL",
"connector_id": "tMongoDBOutput_1",
"component_start_time_seconds": 1628266578
},
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tMongoDBInput",
"connector_label": "tMongoDBInput_2",
"connector_id": "tMongoDBInput_2",
"target_connector_type": "tLogRow",
"target_label": "tLogRow_2",
"target_id": "tLogRow_2",
"connection_type": "row7",
"component_start_time_seconds": 1628267070,
"component_connection_rows_total": 2158754
},
{
"engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
"workspace_id": "6093f6a36f01d639680d31cd",
"task_id": "610d3b7da8abfa32ffb6506b",
"task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
"operator": "yvinqueur",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tLogRow",
"connector_label": "tLogRow_2",
"connector_id": "tLogRow_2",
"component_start_time_seconds": 1628267070
}
],
"limit": 50,
"offset": 0,
"total": 7
}
RawComponentMetric
Name | Description | Type | Attributes and examples |
---|---|---|---|
engine_id | Engine ID | string Required |
ee7bd2c6-7341-42e2-96b2-bbee676c006d |
workspace_id | Workspace ID | string Required |
615ff8ce04ca1e5c0187888d |
task_id | Task ID | string Required |
615ff905ee484c7e15cc0c35 |
task_execution_id | Task execution ID | string Required |
b1b23751-046f-43a2-b47c-a11d0bfaaa5b |
operator | Operator | string Optional |
admin |
operator_type | Operator type | string Optional |
HUMAN |
process_id | Process ID | string Required |
0329f8d4-1c69-3372-9233-d38ac6ef03a8 |
job_name | Job name | string Required |
MainJob |
pid | PID | string Required |
PP0pFQ |
father_pid | Father PID | string Required |
20210806181617_2Y68g |
root_pid | Root PID | string Required |
20210806181617_2Y68g |
connector_type | Connector type | string Required |
tMongoDBConnection |
connector_label | Connector label | string Required |
tMongoDBConnection_1 |
connector_id | Connector ID | string Required |
tMongoDBConnection_1 |
target_connector_type | Target connector type | string Optional |
tStatCatcher |
target_label | Target connector label | string Optional |
tStatCatcher_1 |
target_id | Target connector ID | string Optional |
tStatCatcher_1 |
connection_type | Connection type | string Optional |
output |
connection_name | Connection name | string Optional |
row1 |
component_start_time_seconds | Component start time seconds | number Required |
1628266578 |
component_connection_rows_total | Component connection rows total | number Optional |
5000000 |
component_execution_duration_milliseconds | Component execution duration milliseconds | number Optional |
491585 |
stacktrace | Contains the component stacktrace (only if an error occurs while executing the component) | string Optional |
NullPointer |
{
"engine_id": "ee59bcf2-cb98-48f7-8fcb-36fddfaf8670",
"workspace_id": "61273932d0366133d05729b7",
"task_id": "612846359a0ac71b8f3ed4ea",
"task_execution_id": "43f21d52-497f-4a77-bdbd-2b0ece33e4ee",
"operator": "admin",
"operator_type": "HUMAN",
"process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
"job_name": "MainJob",
"pid": "20210806181617_2Y68h",
"father_pid": "20210806181617_2Y68h",
"root_pid": "20210806181617_2Y68h",
"connector_type": "tStatCatcher",
"connector_label": "tStatCatcher_1",
"connector_id": "tStatCatcher_1",
"target_connector_type": "tLogRow",
"target_label": "tLogRow_2",
"target_id": "tLogRow_2",
"connection_type": "output",
"connection_name": "row1",
"component_start_time_seconds": 1630029441,
"component_connection_rows_total": 100,
"component_execution_duration_milliseconds": 20,
"stacktrace": "NullPointer"
}
ErrorResponse
Name | Description | Type | Attributes and examples |
---|---|---|---|
status | Status Code | integer Required |
INT32 400 |
message | Error Message | string Required |
Task execution with ID '2b252f47-ccfd-4c13-9c4a-d864b599086f not found. |
requestId | Request ID | string Required |
16fefb53-035a-4249-af9d-f80a3b47b132 |
{
"status": 400,
"message": "Parameter 'invalid_field' is not valid field.",
"requestId": "16fefb53-035a-4249-af9d-f80a3b47b132"
}