Using component observability metrics to build reports for task runs

Use POST at the /monitoring/observability/metrics/component endpoint to build reports about task runs.

Get the ID of the task you need to analyze.

The task ID is available on the Task details page in Talend Cloud Management Console. If you need to get this ID via API, follow this example.

Issue the following API request to find the historical runs of this task during a given period:

method: GET
endpoint: 
      https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions?from={start_timestamp_ms}&to={end_timestamp_ms}
headers: {
 "Authorization": "Bearer <your_personal_or_service_account_access_token>"
}
payload: N/A

From the response, count the number of runs and find those that failed, if any.

Issue the following API request to find the execution duration of a specific type of component in each of the task runs:

method: POST
endpoint: 
      https://api.<env>.cloud.talend.com/monitoring/observability/metrics/component
headers: {
 "Authorization": "Bearer <your_personal_or_service_account_access_token>"
}
payload: {
    "startTime": "job start time", 
    "endTime": "job end time", 
    "environmentId": "environment id value",
    "sortBy": "component_execution_duration_milliseconds",
    "sortOrder": "desc";
    "filters": [
        {
            "field": "task_id ",
            "operator": "in", 
            "value": ["ID_of_the_task_to_analyze"]
        },
        {
            "field": "connector_type ",
            "operator": "in", 
            "value": ["a_component_type_such_as_tMongoDBConnection"]
        }
    ]
}

Analyze the response to identify how the execution duration of this type of components varies from one task run to another.

This report helps you understand the circumstances in which a component changes its behavior among task runs.

Below is an example of the response of the POST request at /monitoring/observability/metrics/component.

{
  "items": [
    {
      "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "engine_type": "CLOUD",
      "engine_version": "2.10.7",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "xxx",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tMongoDBConnection",
      "connector_label": "tMongoDBConnection_1",
      "connector_id": "tMongoDBConnection_1",
      "component_start_time_seconds": 1628266578
    },
    {
      "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "engine_type": "CLOUD",
      "engine_version": "2.10.7",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "xxx",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tRowGenerator",
      "connector_label": "tRowGenerator_1",
      "connector_id": "tRowGenerator_1",
      "target_connector_type": "tFlowMeter",
      "target_label": "tFlowMeter_row1",
      "target_id": "tFlowMeter_row1",
      "connection_type": "row1",
      "connection_name": "output",
      "component_start_time_seconds": 1628266578,
      "component_connection_rows_total": 5000000,
      "component_execution_duration_milliseconds": 491585
    }
  ],
  "limit": 50,
  "offset": 0,
  "total": 7
}

Previous section: Troubleshooting a hanging task run