Fetching run logs periodically for analysis

Use the monitoring endpoints to fetch run logs of a given period.

Before you begin

  • Generate an access token:
    • For users, generate a personal access token. For further information, see Generating a Personal Access Token.
    • For service accounts, generate a service account token. For further information about how to generate a service account token, see Generating a service account token. Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.{env}.cloud.talend.com/security/oauth/token.

About this task

In this example, Talend API Tester is used to issue API requests. For further information about Talend API Tester, see Talend Cloud API Tester User Guide.

Procedure

  1. Open Talend API Tester in your browser and select POST from the Method list.

  2. In the field next to the Method drop-down list, enter the endpoint to be used: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs

  3. Click the Add query parameter button to add the startTime and the endTime parameters. These parameters allow you to limit the query to a specific time range of your logs.

    Note that the date value must follow this format:

    yyyy-mm-ddThh:mm:ssZ
    

    For example, 2021-09-01T07:01:45Z.

  4. In the HEADERS area, click Add header and in the name field that is displayed, enter Authorization and in the value field, enter Bearer and your personal access token or service account token. Enter a whitespace to separate Bearer and your token.

  5. Click Send to issue your request.

    The response contains the size of the generated full log and the token to access it. For example,

    {
    "size": the_full_log_size,
    "token": "the_generated_token"
    }
    
  6. Click the Save button to save this request in your repository on Talend API Tester. When you save the request, you need to give it a name. In this example, this request is named as generate-full-log-and-token.

  7. Create a new request to issue a POST call.

  8. Select POST from the Method list.

  9. In the field next to the Method drop-down list, enter the endpoint to be used: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs

  10. In the Body field, enter the one-time token you obtained from the previous request.

    On Talend API Tester, you can use the expression builder to enter an expression, instead of the static token, to dynamically get the latest token generated by the previous request. Note that this request must have been saved in your repository on Talend API Tester. In this example, this saved request is generate-full-log-and-token.

    Post Customer log

    Here is an example of what the expression builder may look like: Expression Builder

  11. Click Send to issue your request.

    The response contains the download URL of the generated full log and the status about whether this log is ready to be downloaded.

    If the status shows that the log is not READY yet, issue this call again. When the log is ready, use the returned URL to download it.

    For example,

    {
    "presignedURL": "URL_to_download_logs",
    "status": "READY"
    }
    

You can then integrate these API requests in your own monitoring system to periodically generate and download the full log of a given time range for analytic purpose.

Only one run is used in this scenario for demonstration purposes. If you want to analyze all the runs of a task, you can use a GET method at the following endpoint to get the IDs of these runs:

 https://api.<your_environment>.cloud.talend.com/processing/executables/tasks/{taskId}/executions

Then pass each of the IDs dynamically to the /executions/{runId}/logs and the /executions/{runId}/logs/status endpoints to prepare the logs to be used.

After all, you can develop a program, such as a Talend Job, to automate the whole process in your monitoring system.

API Job for customer logs