Monitoring task runs by accessing only relevant logs

Obtain task runs of a given period in a specific environment for monitoring purposes.

Before you begin

  • Generate an access token:
    • For users, generate a personal access token. For further information, see Generating a Personal Access Token.
    • For service accounts, generate a service account token. For further information about how to generate a service account token, see Generating a service account token. Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.{env}.cloud.talend.com/security/oauth/token.

About this task

In this section, the following API call is firstly issued:

method: POST
endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks/executions
headers: {
 "Content-Type": "application/json",
 "Authorization": "Bearer <your_personal_access_token_or_service_account_token>"
}
payload: {
  "lastDays": "7",
  "status": "execution_failed",
  "environmentId": "id_of_the_environment_to_be_monitored"
  "tag": [
    "critical"
  ]
}

It is implemented in Talend API Tester for demonstration purposes.

Procedure

  1. Open Talend API Tester in your browser and select POST from the Method list.

  2. In the field next to the Method drop-down list, enter the endpoint to be used:

    In this example, the status, the lastDays, the tag, and the environmentId filters are used to retrieve all the failed critical runs of last 7 days in the given environment.

    The environmentId parameter identifies the environment from which you get the task runs. You can obtain it through API requests or find it in Talend Cloud Management Console. The tag parameter is used to select the failed runs that are tagged with critical.

    • In Talend Cloud Management Console, select the environment from the environment list on the Environment page. The ID of this environment appears in the URL of the current page.

      Find Environment ID

  3. Issue this API request to get a list of the failed critical runs in this environment.

  4. Use each of these run IDs to issue the following API request to obtain their logs.

    method: GET
endpoint: 
  https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/log
headers: {
   "Authorization": "Bearer <personal_access_token_or_service_account_token>"
                  }
payload: N/A

Results

You get each of these logs in the response of the API call. If you want to generate log files to download, see this example: Fetching run logs periodically for analysis

Next section: Monitoring a specific task without downloading logs