Fetching run logs periodically for analysis

Use the monitoring endpoints to fetch run logs of a given period.

Use the POST method at /executions/{runId}/logs to generate the full log of a given period of a run as well as the token to access this log. This token is valid for one hour after being generated.
Use the POST method at /executions/{runId}/logs/status to check whether this log is ready for download and generate a download URL accordingly.

Before you begin

Generate an access token:
- For users, generate a personal access token. For further information, see Generating a Personal Access Token.
- For service accounts, generate a service account token. For further information about how to generate a service account token, see Generating a service account token. Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.{env}.cloud.talend.com/security/oauth/token.

You must have the Author, the Viewer and the Executor permissions for the workspace to which the task to be monitored belongs.
You must know the ID of the task run you need to monitor.

This ID is available on the Task execution log page, reading as Task execution ID. Or you can obtain this ID via a GET call from the /executables/tasks/{taskId}/executions endpoint.

About this task

In this example, Talend API Tester is used to issue API requests. For further information about Talend API Tester, see Talend Cloud API Tester User Guide.

Procedure

Open Talend API Tester in your browser and select POST from the Method list.
In the field next to the Method drop-down list, enter the endpoint to be used: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs
Click the Add query parameter button to add the startTime and the endTime parameters. These parameters allow you to limit the query to a specific time range of your logs.

Note that the date value must follow this format:
```
yyyy-mm-ddThh:mm:ssZ
```
For example, 2021-09-01T07:01:45Z.
In the HEADERS area, click Add header and in the name field that is displayed, enter Authorization and in the value field, enter Bearer and your personal access token or service account token. Enter a whitespace to separate Bearer and your token.
To ensure the completeness of your log, check that the run for which you need the log has completed and then wait for 30 seconds before sending the API call.
Click Send to issue your request.

The response contains the size of the generated full log and the token to access it. For example,
```
{
"size": the_full_log_size,
"token": "the_generated_token"
}
```
Click the Save button to save this request in your repository on Talend API Tester. When you save the request, you need to give it a name. In this example, this request is named as generate-full-log-and-token.
Create a new request to issue a POST call.
Select POST from the Method list.
In the field next to the Method drop-down list, enter the endpoint to be used: https://api.<your_environment>.cloud.talend.com/monitoring/executions/<runId>/logs/status
In the Body field, enter the one-time token you obtained from the previous request.

On Talend API Tester, you can use the expression builder to enter an expression, instead of the static token, to dynamically get the latest token generated by the previous request. Note that this request must have been saved in your repository on Talend API Tester. In this example, this saved request is generate-full-log-and-token.

Here is an example of what the expression builder may look like:
Click Send to issue your request.

The response contains the download URL of the generated full log and the status about whether this log is ready to be downloaded.

If the status shows that the log is not READY yet, issue this call again. When the log is ready, use the returned URL to download it.

For example,
```
{
"presignedURL": "URL_to_download_logs",
"status": "READY"
}
```

You can then integrate these API requests in your own monitoring system to periodically generate and download the full log of a given time range for analytic purpose.

Only one run is used in this scenario for demonstration purposes. If you want to analyze all the runs of a task, you can use a GET method at the following endpoint to get the IDs of these runs:

 https://api.<your_environment>.cloud.talend.com/processing/executables/tasks/{taskId}/executions

Then pass each of the IDs dynamically to the /executions/{runId}/logs and the /executions/{runId}/logs/status endpoints to prepare the logs to be used.

After all, you can develop a program, such as a Qlik Talend Job, to automate the whole process in your monitoring system.

API Job for customer logs

Previous section: Getting a task run log for live monitoring