Monitoring a specific task without downloading logs

Obtain run metrics of a specific task without having to generate or download its logs.

Before you begin

Generate an access token:
- For users, generate a personal access token. For further information, see Generating a Personal Access Token.
- For service accounts, generate a service account token. For further information about how to generate a service account token, see Generating a service account token. Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.{env}.cloud.talend.com/security/oauth/token.

The user or the service account to be used must have the Viewer permission for the workspace to which the task to be monitored belongs.
You must know the ID of this workspace. Get this ID either via an API request or through Talend Cloud Management Console:
- API: use GET at the https://api.<your_environment>.cloud.talend.com/orchestration/workspaces.
- Talend Cloud Management Console: the ID of the workspace appears in the URL of the Edit workspace page.

You must know the ID of the task you need to monitor.

In Talend Cloud Management Console, this ID is displayed on the dedicated page of the task to be monitored. For example:

Task ID

Or you can obtain task IDs via the following API request:

method: GET
endpoint: https://api.<env>.cloud.talend.com/orchestration/executables/tasks
headers: {
  "Content-Type": "application/json",
  "Authorization": "Bearer <your_personal_access_token_or_service_account_token>"
          }
query parameters: {
   "workspaceId": "id_of_the_workspace_of_the_task_to_be_monitored"
                  }

About this task

In this section, the following API call is firstly issued:

method: GET
endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions
headers: {
 "Content-Type": "application/json",
 "Authorization": "Bearer <your_personal_access_token_or_service_account_token>"
}
query parameters: {
  "lastDays": "1",
  "status": "execution_failed"
}

It is implemented in Talend API Tester for demonstration purposes.

Procedure

Open Talend API Tester in your browser and select GET from the Method list.
In the field next to the Method drop-down list, enter the endpoint to be used:

In this example, the status and the lastDays parameters are used to retrieve all the failed runs on a previous day of a specific task.
Issue this API request to get a list of the failed runs.

Use each of these run IDs to issue the following API request to their observability metrics:

method: GET
endpoint: 
  https://api.<env>.cloud.talend.com/monitoring/observability/executions/{runId}/component
headers: {
   "Authorization": "Bearer <personal_access_token_or_service_account_token>"
                  }
payload: N/A

Analyze the response to identify the cause of each failure. For example, if you see a component does not have the component_execution_duration_milliseconds field, this component could be hanging.

The following section presents more examples about how to use this metrics.

Results

Below is an example of the response of the GET request at /monitoring/observability/executions/{runId}/component.

{
    "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
    "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
    "engine_type": "CLOUD",
    "engine_version": "2.10.8",
    "workspace_id": "61273932d0366133d05729b7",
    "task_id": "612739e79a0ac71b8f3ed4dd",
    "task_execution_id": "947e3e2f-d199-4988-a5ab-14ceb36c80f3",
    "artifact_id": "612739e79a0ac71b8f3ed4db",
    "artifact_name": "job_with_rejected_rows",
    "artifact_version": "0.1.0.20212608065119",
    "start_time": "2021-08-26T06:53:30.127Z",
    "finish_time": "2021-08-26T06:53:35.361Z",
    "rows_rejected": 1,
    "operator": "admin",
    "operator_type": "HUMAN",
    "processes": [
                    {
                      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
                      "job_name": "MainJob",
                      "pid": "20210806181617_2Y68h",
                      "father_pid": "20210806181617_2Y68h",
                      "root_pid": "20210806181617_2Y68h"
                    }
                 ],
    "metrics": {
                "items": [
                          {
                            "pid": "20210806181617_2Y68h",
                            "connector_type": "tMongoDBConnection",
                            "connector_label": "tMongoDBConnection_1",
                            "connector_id": "tMongoDBConnection_1",
                            "component_start_time_seconds": 1628266578
                           },
                           {
                            "pid": "20210806181617_2Y68h",
                            "connector_type": "tRowGenerator",
                            "connector_label": "tRowGenerator_1",
                            "connector_id": "tRowGenerator_1",
                            "target_connector_type": "tFlowMeter",
                            "target_label": "vFlowMeter_row1",
                            "target_id": "vFlowMeter_row1",
                            "component_start_time_seconds": 1628266578,
                            "component_connection_rows_total": 5000000,
                            "component_execution_duration_milliseconds": 491585
                            },
                            {
                            "pid": "20210806181617_2Y68h",
                            "connector_type": "tFlowMeter",
                            "connector_label": "vFlowMeter_row1",
                            "connector_id": "vFlowMeter_row1",
                            "component_start_time_seconds": 1628266578
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tFlowMeter",
                             "connector_label": "vFlowMeter_row1",
                             "connector_id": "vFlowMeter_row1",
                             "target_connector_type": "tMongoDBOutput",
                             "target_label": "Insert from SQL",
                             "target_id": "tMongoDBOutput_1",
                             "component_start_time_seconds": 1628266578,
                             "component_connection_rows_total": 5000000,
                             "component_execution_duration_milliseconds": 491605
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tMongoDBOutput",
                             "connector_label": "Insert from SQL",
                             "connector_id": "tMongoDBOutput_1",
                             "component_start_time_seconds": 1628266578
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tMongoDBInput",
                             "connector_label": "tMongoDBInput_2",
                             "connector_id": "tMongoDBInput_2",
                             "target_connector_type": "tLogRow",
                             "target_label": "tLogRow_2",
                             "target_id": "tLogRow_2",
                             "component_start_time_seconds": 1628267070,
                             "component_connection_rows_total": 2158754
                            },
                            {
                             "pid": "20210806181617_2Y68h",
                             "connector_type": "tLogRow",
                             "connector_label": "tLogRow_2",
                             "connector_id": "tLogRow_2",
                             "component_start_time_seconds": 1628267070
                            }
                           ],
                  "limit": 50,
                  "offset": 0,
                  "total": 7
                  }
}

Previous section: Monitoring task runs by accessing only relevant logs Next section: Getting a task run log for live monitoring