Monitoring a specific task without downloading logs

Obtain run metrics of a specific task without having to generate or download its logs.

Before you begin

  • Generate an access token:
    • For users, generate a personal access token. For further information, see Generating a Personal Access Token.
    • For service accounts, generate a service account token. For further information about how to generate a service account token, see Generating a service account token. Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.{env}.cloud.talend.com/security/oauth/token.

About this task

In this section, the following API call is firstly issued:

method: GET
endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions
headers: {
 "Content-Type": "application/json",
 "Authorization": "Bearer <your_personal_access_token_or_service_account_token>"
}
payload: {
  "lastDays": "1",
  "status": "execution_failed",
  "workspaceId": "id_of_the_workspace"
}

It is implemented in Talend API Tester for demonstration purposes.

Procedure

  1. Open Talend API Tester in your browser and select GET from the Method list.

  2. In the field next to the Method drop-down list, enter the endpoint to be used:

    In this example, the status, the lastDays and the workspaceId parameters are used to retrieve all the failed runs on a previous day of a specific task in the given workspace.

    Get specific task runs

  3. Issue this API request to get a list of the failed runs.

  4. Use each of these run IDs to issue the following API request to their observability metrics:

    method: GET
    endpoint: 
      https://api.<env>.cloud.talend.com/monitoring/observability/executions/{runId}/component
    headers: {
       "Authorization": "Bearer <personal_access_token_or_service_account_token>"
                      }
    payload: N/A
    
  5. Analyze the response to identify the cause of each failure. For example, if you see a component does not have the component_execution_duration_milliseconds field, this component could be hanging.

    The following section presents more examples about how to use this metrics.

Results

Below is an example of the response of the GET request at /monitoring/observability/executions/{runId}/component.

{
    "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
    "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
    "engine_type": "CLOUD",
    "engine_version": "2.10.8",
    "workspace_id": "61273932d0366133d05729b7",
    "task_id": "612739e79a0ac71b8f3ed4dd",
    "task_execution_id": "947e3e2f-d199-4988-a5ab-14ceb36c80f3",
    "artifact_id": "612739e79a0ac71b8f3ed4db",
    "artifact_name": "job_with_rejected_rows",
    "artifact_version": "0.1.0.20212608065119",
    "start_time": "2021-08-26T06:53:30.127Z",
    "finish_time": "2021-08-26T06:53:35.361Z",
    "rows_rejected": 1,
    "operator": "admin",
    "operator_type": "HUMAN",
    "processes": [
                    {
                      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
                      "job_name": "MainJob",
                      "pid": "20210806181617_2Y68h",
                      "father_pid": "20210806181617_2Y68h",
                      "root_pid": "20210806181617_2Y68h"
                    }
                 ],
    "metrics": {
                "items": [
                          {
                            "pid": "20210806181617_2Y68h",
                            "connector_type": "tMongoDBConnection",
                            "connector_label": "tMongoDBConnection_1",
                            "connector_id": "tMongoDBConnection_1",
                            "component_start_time_seconds": 1628266578
                           },
                           {
                            "pid": "20210806181617_2Y68h",
                            "connector_type": "tRowGenerator",
                            "connector_label": "tRowGenerator_1",
                            "connector_id": "tRowGenerator_1",
                            "target_connector_type": "tFlowMeter",
                            "target_label": "vFlowMeter_row1",
                            "target_id": "vFlowMeter_row1",
                            "component_start_time_seconds": 1628266578,
                            "component_connection_rows_total": 5000000,
                            "component_execution_duration_milliseconds": 491585
                            },
                            {
                            "pid": "20210806181617_2Y68h",
                            "connector_type": "tFlowMeter",
                            "connector_label": "vFlowMeter_row1",
                            "connector_id": "vFlowMeter_row1",
                            "component_start_time_seconds": 1628266578
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tFlowMeter",
                             "connector_label": "vFlowMeter_row1",
                             "connector_id": "vFlowMeter_row1",
                             "target_connector_type": "tMongoDBOutput",
                             "target_label": "Insert from SQL",
                             "target_id": "tMongoDBOutput_1",
                             "component_start_time_seconds": 1628266578,
                             "component_connection_rows_total": 5000000,
                             "component_execution_duration_milliseconds": 491605
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tMongoDBOutput",
                             "connector_label": "Insert from SQL",
                             "connector_id": "tMongoDBOutput_1",
                             "component_start_time_seconds": 1628266578
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tMongoDBInput",
                             "connector_label": "tMongoDBInput_2",
                             "connector_id": "tMongoDBInput_2",
                             "target_connector_type": "tLogRow",
                             "target_label": "tLogRow_2",
                             "target_id": "tLogRow_2",
                             "component_start_time_seconds": 1628267070,
                             "component_connection_rows_total": 2158754
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tLogRow",
                             "connector_label": "tLogRow_2",
                             "connector_id": "tLogRow_2",
                             "component_start_time_seconds": 1628267070
                            }
                           ],
                  "limit": 50,
                  "offset": 0,
                  "total": 7
                  }
}