Observability Metrics

Retrieve execution metrics

Endpoints

Global security

These security schemes apply to the entire API

Security scheme

This scheme can be referenced across the API

Public
This Bearer authentication can be set in the Authorization header of your requests. Authentication tokens and Personal Access Tokens are supported. Personal Access Tokens can be generated in the Profile Preferences page of the Talend Cloud Portal.
Bearer authentication
Name Description

Get component metrics of task runs

GET /monitoring/observability/executions/{executionId}/component

This endpoint is used to get component metrics of a given task run.

To retrieve your metrics data accurately, the process may vary in duration.

Security

Request

Path variables
Name Description Type Attributes and examples
executionId Task Execution ID string Required
Query parameters
Name Description Type Attributes and examples
offset The start position of the pagination of the entries, greater or equal to 0 integer Optional
INT32
limit Number of entries to return. The default value is 50 and the maximum value is 200 integer Optional
50
INT32
sortBy The field by which the returned entries are sorted. The default value is component_start_time_seconds string Optional
component_start_time_seconds
sortOrder

The way in which the returned logs are sorted, desc or asc

Whether the order is ascending or descending, a null or an empty field is always placed at the end of the list to be returned in the response

string Optional
asc
Headers
Name Description Type Attributes and examples
talend-version string Optional

Response

200Status 200
OK
Body
ExecutionComponentMetrics
400Status 400
400
Body
ErrorResponse
Status 400 application/json
{
    "status": 400,
    "message": "Task execution ID is not valid id.",
    "requestId": "f1f6d388-b114-4d02-8d90-da2c8d1af60f"
}
401Status 401
401
Body
ErrorResponse
Status 401 application/json
{
    "status": 401, 
    "message": "Unauthorized"
}
403Status 403
403
Body
ErrorResponse
Status 403 application/json
{
    "status": 403, 
    "message": "Forbidden",
    "requestId": "8c7c7de7-0a9a-45be-9e28-f0a34e6007fd"
}
404Status 404
404
Body
ErrorResponse
Status 404 application/json
{
    "status": 404,
    "message": "Task execution with ID '83503269-d6a1-437f-bf91-d38ad74a1ecf' not found.",
    "requestId": "281f0387-9e88-4879-835d-30c1f3509af3"
}
500Status 500
500
Body
ErrorResponse
Status 500 application/json
{
    "status": 500,
    "message": "Internal Server Error",
    "requestId": "2b6e98ae-66dd-4895-8f04-dc2f85617855"
}

Search raw component metrics

POST /monitoring/observability/metrics/component

This Public API is mean to query raw component metrics freely.

To retrieve your metrics data accurately, the process may vary in duration.

Request

Headers
Name Description Type Attributes and examples
talend-version string Optional
Body
ComponentQueryRequest

Response

200Status 200
Status 200
Body
RawComponentMetricPage
400Status 400
400
Body
ErrorResponse
Status 400 application/json
{
    "status": 400,
    "message": "Task execution ID is not valid id.",
    "requestId": "f1f6d388-b114-4d02-8d90-da2c8d1af60f"
}
401Status 401
401
Body
ErrorResponse
Status 401 application/json
{
    "status": 401, 
    "message": "Unauthorized"
}
403Status 403
403
Body
ErrorResponse
Status 403 application/json
{
    "status": 403, 
    "message": "Forbidden",
    "requestId": "8c7c7de7-0a9a-45be-9e28-f0a34e6007fd"
}
500Status 500
500
Body
ErrorResponse
Status 500 application/json
{
    "status": 500,
    "message": "Internal Server Error",
    "requestId": "2b6e98ae-66dd-4895-8f04-dc2f85617855"
}

ComponentQueryRequest

Object
Name Description Type Attributes and examples
environmentId Environment ID string Required
startTime

Start time of metrics generation and collection. This time corresponds approximately to when your Job task starts to work.

  • Using ISO format yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
string Required
endTime

End time of metrics generation and collection. This time corresponds approximately to when this component in your Job task finishes its work

  • Using ISO format yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
  • endTime must be greater than or equals to startTime
string Required
limit Page size, 1~200 number Optional
50
offset Page offset, >=0 number Optional
sortBy Sort by field string Optional
component_start_time_seconds
sortOrder Sort order string Optional
asc
filters array of ComponentFieldFilter Optional
Datatype details
Type Description Attributes and examples
array Field filters
ComponentFieldFilter
ComponentQueryRequest
{
    "environmentId": "6093f6a36f01d639680d31cd", 
    "startTime": "2021-09-08T06:58:04.604Z", 
    "endTime": "2021-09-08T06:58:06.463Z", 
    "limit": 50, 
    "offset": 0, 
    "sortBy": "component_start_time_seconds", 
    "sortOrder": "asc", 
    "filters": [
       {
            "field": "engine_id", 
            "operator": "in", 
            "value": ["a60bb1c0-7669-407f-9326-138af05da18a"]
        },
        {
            "field": "component_start_time_seconds", 
            "operator": "gte", 
            "value": 1628266578
        }
    ]
}

ComponentFieldFilter

Object
Name Description Type Attributes and examples
field Field filter name string Required
operator

Field filter operator

  • Field component_start_time_seconds supports gt, gte, lt, lte, others support in
  • Can’t use more than 2 operators
  • Can’t use any 2 same operators
  • Can’t use gt together with gte
  • Can’t use lttogether with lte
  • When use gt/gte and lt together, value of lt must be greater than value of gt/gte
  • When use gte and lt together, value of lt must be greater than value of gte
  • When use gte and lte together, value of lte must be greater than or equals to value of gte
string Required
value

Field filter value

For operator in requires an array, like

  • "value" : ["value"]
  • "value" : ["value1", "value2"]

For operator gt, gte, lt, lte requres a number(or a string of number), like

  • "value" : 1628266578
  • "value" : 1628266578.78
  • "value" : "1628266578"
  • "value" : "1628266578.78"
ObjectRequired
ComponentFieldFilter
{
    "field": "engine_id", 
    "operator": "in", 
    "value": ["a60bb1c0-7669-407f-9326-138af05da18a"]
}

ExecutionComponentMetrics

Object
Name Description Type Attributes and examples
engine_id Engine ID string Required
workspace_id Workspace ID string Required
task_id Task ID string Required
task_execution_id Task execution ID string Required
artifact_id Artifact ID string Required
artifact_name Artifact name string Optional
artifact_version Artifact version string Required
start_time Jon run start time string Required
finish_time Job run finish time string Optional
rows_rejected Number of rows rejected number Optional
operator Operator string Optional
operator_type Operator type string Optional
processes array of ProcessMetric Required
Datatype details
Type Description Attributes and examples
array Job processes
ProcessMetric
metrics ComponentMetricPage Required
ExecutionComponentMetrics
{
    "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
    "workspace_id": "61273932d0366133d05729b7",
    "task_id": "612739e79a0ac71b8f3ed4dd",
    "task_execution_id": "947e3e2f-d199-4988-a5ab-14ceb36c80f3",
    "artifact_id": "612739e79a0ac71b8f3ed4db",
    "artifact_name": "job_with_rejected_rows",
    "artifact_version": "0.1.0.20212608065119",
    "start_time": "2021-08-26T06:53:30.127Z",
    "finish_time": "2021-08-26T06:53:35.361Z",
    "rows_rejected": 1,
    "operator": "admin",
    "operator_type": "HUMAN",
    "processes": [
        {
            "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
            "job_name": "MainJob",
            "pid": "20210806181617_2Y68h",
            "father_pid": "20210806181617_2Y68h",
            "root_pid": "20210806181617_2Y68h"
        }
    ],
    "metrics": {
        "items": [
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tMongoDBConnection",
                "connector_label": "tMongoDBConnection_1",
                "connector_id": "tMongoDBConnection_1",
                "component_start_time_seconds": 1628266578
            },
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tRowGenerator",
                "connector_label": "tRowGenerator_1",
                "connector_id": "tRowGenerator_1",
                "target_connector_type": "tFlowMeter",
                "target_label": "vFlowMeter_row1",
                "target_id": "vFlowMeter_row1",
                "component_start_time_seconds": 1628266578,
                "component_connection_rows_total": 5000000,
                "component_execution_duration_milliseconds": 491585
            },
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tFlowMeter",
                "connector_label": "vFlowMeter_row1",
                "connector_id": "vFlowMeter_row1",
                "component_start_time_seconds": 1628266578
            },
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tFlowMeter",
                "connector_label": "vFlowMeter_row1",
                "connector_id": "vFlowMeter_row1",
                "target_connector_type": "tMongoDBOutput",
                "target_label": "Insert from SQL",
                "target_id": "tMongoDBOutput_1",
                "component_start_time_seconds": 1628266578,
                "component_connection_rows_total": 5000000,
                "component_execution_duration_milliseconds": 491605
            },
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tMongoDBOutput",
                "connector_label": "Insert from SQL",
                "connector_id": "tMongoDBOutput_1",
                "component_start_time_seconds": 1628266578
            },
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tMongoDBInput",
                "connector_label": "tMongoDBInput_2",
                "connector_id": "tMongoDBInput_2",
                "target_connector_type": "tLogRow",
                "target_label": "tLogRow_2",
                "target_id": "tLogRow_2",
                "component_start_time_seconds": 1628267070,
                "component_connection_rows_total": 2158754
            },
            {
                "pid": "20210806181617_2Y68h",
                "connector_type": "tLogRow",
                "connector_label": "tLogRow_2",
                "connector_id": "tLogRow_2",
                "component_start_time_seconds": 1628267070
            }
        ],
        "limit": 50,
        "offset": 0,
        "total": 7
    }
}

ProcessMetric

Object
Name Description Type Attributes and examples
process_id Process ID. This is JVM’s UUID string Required
job_name Job Name string Required
pid PID. This is the unique Job identifier for the current run string Required
father_pid ID of the first direct parent Job string Required
root_pid ID of the first root Job if there are several Subjobs string Required
ProcessMetric
{
    "process_id" : "c151b20d-cb9a-34ad-b219-2080761dd193",
    "job_name" : "sub_job_2",
    "pid" : "WTHlkK",
    "father_pid" : "20210827095720_E61Q1",
    "root_pid" : "20210827095720_E61Q1"
}

ComponentMetricPage

Object
Name Description Type Attributes and examples
items array of ComponentMetric Required
Datatype details
Type Description Attributes and examples
array Array of metrics entries about the components in a Job
ComponentMetric
limit Maximum number of entries to be returned on a page in the response number Required
offset The start position of the pagination of the entries, greater or equal to 0 number Required
total Total number of entries in the response number Required
ComponentMetricPage
{
    "items": [
        {
            "pid": "20210827095720_E61Q1", 
            "connector_type": "tParallelize", 
            "connector_label": "tParallelize_1", 
            "connector_id": "tParallelize_1", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "20210827095720_E61Q1", 
            "connector_type": "tJava", 
            "connector_label": "tJava_1", 
            "connector_id": "tJava_1", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "20210827095720_E61Q1", 
            "connector_type": "tJava", 
            "connector_label": "tJava_2", 
            "connector_id": "tJava_2", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "20210827095720_E61Q1", 
            "connector_type": "tRunJob", 
            "connector_label": "tRunJob_1", 
            "connector_id": "tRunJob_1", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "20210827095720_E61Q1", 
            "connector_type": "tRunJob", 
            "connector_label": "tRunJob_2", 
            "connector_id": "tRunJob_2", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "WTHlkK", 
            "connector_type": "tRunJob", 
            "connector_label": "tRunJob_1_subjob2", 
            "connector_id": "tRunJob_1", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "W1yeMB", 
            "connector_type": "tParallelize", 
            "connector_label": "tParallelize_1_subjob1", 
            "connector_id": "tParallelize_1", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "W1yeMB", 
            "connector_type": "tJava", 
            "connector_label": "tJava_1_subjob1", 
            "connector_id": "tJava_1", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "W1yeMB", 
            "connector_type": "tJava", 
            "connector_label": "tJava_2_subjob1", 
            "connector_id": "tJava_2", 
            "component_start_time_seconds": 1630029441
        }, 
        {
            "pid": "OPsysU", 
            "connector_type": "tStatCatcher", 
            "connector_label": "tStatCatcher_1", 
            "connector_id": "tStatCatcher_1", 
            "target_connector_type": "tLogRow", 
            "target_label": "tLogRow_2", 
            "target_id": "tLogRow_2", 
            "component_start_time_seconds": 1630029441, 
            "component_connection_rows_total": 1, 
            "component_execution_duration_milliseconds": 2
        }, 
        {
            "pid": "OPsysU", 
            "connector_type": "tLogRow", 
            "connector_label": "tLogRow_2", 
            "connector_id": "tLogRow_2", 
            "component_start_time_seconds": 1630029441
        }
    ], 
    "limit": 50, 
    "offset": 0, 
    "total": 100
}

ComponentMetric

Object
Name Description Type Attributes and examples
pid Unique Job identifier for the current run string Required
connector_type Connector type string Required
connector_label Connector label string Required
connector_id Connector ID string Required
target_connector_type Target connector type string Optional
target_label Target connector label string Optional
target_id Target connector ID string Optional
component_start_time_seconds Time stamp indicating the component start time (any component in the Job) number Required
component_connection_rows_total Total number of rows treated by a component in the Job number Optional
component_execution_duration_milliseconds Amount of time it takes to execute a Job component (generated for components that are managing rows) number Optional
stacktrace Contains the component stacktrace (only if an error occurs while executing the component) string Optional
ComponentMetric
{
    "pid" : "OPsysU",
    "connector_type" : "tStatCatcher",
    "connector_label" : "tStatCatcher_1",
    "connector_id" : "tStatCatcher_1",
    "target_connector_type" : "tLogRow",
    "target_label" : "tLogRow_2",
    "target_id" : "tLogRow_2",
    "component_start_time_seconds" : 1630029441,
    "component_connection_rows_total" : 5000000,
    "component_execution_duration_milliseconds" : 491585,
    "stacktrace" : "NullPointer"
  }

RawComponentMetricPage

Object
Name Description Type Attributes and examples
items array of RawComponentMetric Required
Datatype details
Type Description Attributes and examples
array Current page data
RawComponentMetric
limit Page size integer Required
INT32
offset Page offset integer Required
INT32
total Total number integer Required
INT32
RawComponentMetricPage
{
  "items": [
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tMongoDBConnection",
      "connector_label": "tMongoDBConnection_1",
      "connector_id": "tMongoDBConnection_1",
      "component_start_time_seconds": 1628266578
    },
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tRowGenerator",
      "connector_label": "tRowGenerator_1",
      "connector_id": "tRowGenerator_1",
      "target_connector_type": "tFlowMeter",
      "target_label": "vFlowMeter_row1",
      "target_id": "vFlowMeter_row1",
      "connection_type": "row1",
      "connection_name": "output",
      "component_start_time_seconds": 1628266578,
      "component_connection_rows_total": 5000000,
      "component_execution_duration_milliseconds": 491585
    },
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tFlowMeter",
      "connector_label": "vFlowMeter_row1",
      "connector_id": "vFlowMeter_row1",
      "component_start_time_seconds": 1628266578
    },
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tFlowMeter",
      "connector_label": "vFlowMeter_row1",
      "connector_id": "vFlowMeter_row1",
      "target_connector_type": "tMongoDBOutput",
      "target_label": "Insert from SQL",
      "target_id": "tMongoDBOutput_1",
      "connection_type": "meterRowrow1",
      "connection_name": "output",
      "component_start_time_seconds": 1628266578,
      "component_connection_rows_total": 5000000,
      "component_execution_duration_milliseconds": 491605
    },
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tMongoDBOutput",
      "connector_label": "Insert from SQL",
      "connector_id": "tMongoDBOutput_1",
      "component_start_time_seconds": 1628266578
    },
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tMongoDBInput",
      "connector_label": "tMongoDBInput_2",
      "connector_id": "tMongoDBInput_2",
      "target_connector_type": "tLogRow",
      "target_label": "tLogRow_2",
      "target_id": "tLogRow_2",
      "connection_type": "row7",
      "component_start_time_seconds": 1628267070,
      "component_connection_rows_total": 2158754
    },
    {
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "yvinqueur",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tLogRow",
      "connector_label": "tLogRow_2",
      "connector_id": "tLogRow_2",
      "component_start_time_seconds": 1628267070
    }
  ],
  "limit": 50,
  "offset": 0,
  "total": 7
}

RawComponentMetric

Object
Name Description Type Attributes and examples
engine_id Engine ID string Required
workspace_id Workspace ID string Required
task_id Task ID string Required
task_execution_id Task execution ID string Required
operator Operator string Optional
operator_type Operator type string Optional
process_id Process ID string Required
job_name Job name string Required
pid PID string Required
father_pid Father PID string Required
root_pid Root PID string Required
connector_type Connector type string Required
connector_label Connector label string Required
connector_id Connector ID string Required
target_connector_type Target connector type string Optional
target_label Target connector label string Optional
target_id Target connector ID string Optional
connection_type Connection type string Optional
connection_name Connection name string Optional
component_start_time_seconds Component start time seconds number Required
component_connection_rows_total Component connection rows total number Optional
component_execution_duration_milliseconds Component execution duration milliseconds number Optional
stacktrace Contains the component stacktrace (only if an error occurs while executing the component) string Optional
RawComponentMetric
{
    "engine_id": "ee59bcf2-cb98-48f7-8fcb-36fddfaf8670", 
    "workspace_id": "61273932d0366133d05729b7", 
    "task_id": "612846359a0ac71b8f3ed4ea", 
    "task_execution_id": "43f21d52-497f-4a77-bdbd-2b0ece33e4ee", 
    "operator": "admin",
    "operator_type": "HUMAN",
    "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8", 
    "job_name": "MainJob", 
    "pid": "20210806181617_2Y68h", 
    "father_pid": "20210806181617_2Y68h", 
    "root_pid": "20210806181617_2Y68h", 
    "connector_type": "tStatCatcher", 
    "connector_label": "tStatCatcher_1", 
    "connector_id": "tStatCatcher_1", 
    "target_connector_type": "tLogRow", 
    "target_label": "tLogRow_2", 
    "target_id": "tLogRow_2", 
    "connection_type": "output", 
    "connection_name": "row1", 
    "component_start_time_seconds": 1630029441, 
    "component_connection_rows_total": 100, 
    "component_execution_duration_milliseconds": 20,
    "stacktrace": "NullPointer"
}

ErrorResponse

Object
Name Description Type Attributes and examples
status Status Code integer Required
INT32
message Error Message string Required
requestId Request ID string Required
ErrorResponse
{
    "status": 400, 
    "message": "Parameter 'invalid_field' is not valid field.",
    "requestId": "16fefb53-035a-4249-af9d-f80a3b47b132"
}
back to top