Release notes

Talend Cloud Management Console and Dynamic Engine solution provides SaaS capabilities and experience on customer-managed infrastructure.

Dynamic Engine serves as a unique infrastructure to simplify task executions, concealing the complexity of classic engines.

You can create and manage your Dynamic Engine instances and their environments from Talend Cloud Management Console user interface or using the Dynamic Engine specific API endpoints.

Your tasks and plans can be processed in a Dynamic Engine infrastructure by the services set up in an associated environment. Only data integration and big data Job tasks are supported.

For more information about Dynamic Engine, see Setting up Dynamic Engines.

New endpoints:

• Dynamic Engine Environments
• Dynamic Engine

The Misfired status has been added to the execution statuses for tasks and plans.

The Misfired status indicates that a task or plan cannot start due to an issue. This is different from failures during executions, which are reflected under the Failed status.

This new status is not applicable to Routes or data service tasks or plans, for which the issues are still shown as Failed.

Keep in mind that the Misfired notification becomes active about one hour after the release. If you had already enabled notifications for Failed before the release, the Misfired notification is automatically enabled for you. After the release, the Failed notification and the Misfired notification are managed independently and no longer affect each other.

A new misfiredExecutionReason object has been added to the JobExecutionStatus and PlanExecutionStatus objects to provide reasons for the Misfired status, when requests are sent to the endpoints using these two objects.

JobExecutionStatus is a new object designed to replace the old JobExecutionStatusV21 object.

Related endpoints:

• Get Task execution status
• Get available Plans executions
• Get Plan executions
• Get Plan Execution status

A new endpoint has been added to help easily delete logs of an execution.

Related endpoints:

• Delete logs of a specified execution

Starting from its R2024-07 release, Cloud Engines will default to Java 11 for executing tasks. If your existing tasks still require Java 8, you can set up a rollback configuration using new endpoints now available from R2024-06. This configuration will ensure your Cloud Engines continue to use Java 8 until you update it using the same endpoints.

However, we strongly encourage you to migrate these tasks to Java 11 as soon as possible. The default Java version for Cloud Engines will continue to evolve, moving to Java 11 and beyond.

To ensure long-term compatibility and take advantage of the advancements in new Java versions, migrating your tasks to newer Java versions is essential.

Related endpoints:

• Retrieve the JVM version rollback configuration for tasks on Cloud Engines
• Set the JVM rollback configuration for tasks executed on Cloud Engines

When configuring how a Route or data service task runs on a Talend Remote Engine cluster using API, you can use the optional parameter deploymentStrategy to determine how this task should be deployed.

• Rolling: Tasks are deployed one by one to the engines of the cluster. This means that for a short time, different engines will be running different versions of the task, therefore you can avoid service interruption.
• Parallel: Tasks are simultaneously deployed to all the engines of the cluster. Before deploying a new version of a task, you will need to undeploy the currently running version, which will result in service interruption.

Related endpoints:

• Configure Task executions

The Always Proceed to Next Step condition has been added to the existing conditions on a given step to allow your plan to continue the execution flow regardless of the success or failure of individual tasks within this step.

This new condition enhances flexibility in plan design and increases resilience in plan executions, ensuring subsequent steps are executed irrespective of the outcomes of each individual task within a given step.

Related endpoints:

• Create Plan
• Update plan

New Monitoring APIs have been added to help you easily search for historical executions by filtering on a wide range of execution fields. These fields include the types of the execution like a plan or a task, the category of an artifact such as a standard Job, a Route, or a Pipeline, and the runtime configuration like the engine, the workspace, or the environment.

Additionally, These APIs allow you to conveniently download the results in a CSV file for further processing.

Related endpoint:

• Execution History Search

The Search raw component metrics endpoint has been improved to be more accurate when searching for component metrics data by the start or the end time of the components.

The updated filters are:

• startTime: start time of metrics generation and collection. This time corresponds approximately to when your Job task starts to work.
• endTime: end time of metrics generation and collection. This time corresponds approximately to when this component in your Job task finishes its work
• component_start_time_seconds: this filter has been deprecated

Related endpoint:

• Search raw component metrics

When unpairing a Remote Engine using the Unpair Remote Engine API endpoint, you can use new options to maintain uninterrupted your running data service tasks and Route tasks:

• keepOsgiDeployments: set this option to true to keep OSGi services running
• keepMicroserviceDeployments: set this option to true to keep microservices running

These options offer greater flexibility and control for your unpairing process.

Related endpoint:

• Unpair Remote Engine

Consistent endeavor is continuously being carried out to address API discrepancy issues to help you efficiently develop your API solutions.

Highlighted fixes:

• Get Plan configuration and Configure Plan execution: the API documentation is corrected to confirm that the Runtime attribute is not required in the response or the request body.
• Get Plan details: a BaseArtifactVersion type instead of artifactVersion is now used in the payload.

When deleting a run profile, use the newRunProfileId query parameter to specify the run profile that replaces the deleted one.

In addition, a list of tasks using the run profile to be deleted is returned:

• If you do not provide a new run profile ID for the replacement, the deletion fails because the tasks on this list still use this run profile and you have not provided an alternative.
• If you provide a new run profile ID, the deletion succeeds. Then you need to update the tasks on that list with the new run profile ID.

Related endpoints:

• Delete engine run profile

When filtering task executions using the from and the to filters, the executions to be included in the response are:

• Executions that started in the specified timeframe between from and to.
• Executions that ended in the specified timeframe between from and to.
• Executions that started and ended in the specified timeframe between from and to.
• Executions that ran throughout the entire specified timeframe between from and to.

Updated endpoint:

• Get available task executions
• Get Tasks Executions

Maximum Transaction Per Minute(TPM) rates have been increased to more effectively limit the concurrent API calls from a single IP address:

• Monitoring: 600
• Orchestration: 1800
• Processing: 1800
• Seats and subscription: 60
• Workspace permissions: 600

These limits serve to safeguard against potential misuse or attacks, ensure equitable access to resources, better handle sudden spikes, and enable smooth functioning under diverse scenarios, ranging from standard to exceptional.

Related endpoints:

• Monitoring:
  • Execution Logs
  • Observability Metrics
• Orchestration
• Processing
• Workspace Permissions

A new endpoint has been added to help you easily retrieve details of your license, such as:

• Number of tokens you acquired
• How they are allocated for various usage scenarios
• Your current subscription options

Related endpoint:

• Seats and Subscription

When publishing the new version of an artifact, by default, all the tasks that meet the following conditions are automatically updated, no matter in which workspace this publishing happens:

• they are configured with Always use the latest artifact version
• they are using an artifact that has the same name as the artifact being published

You can now protect a workspace and therefore its tasks from this kind of unintended update, by setting the protectedArtifactUpdate parameter to be true, when invoking the workspace update endpoint. With this parameter on, tasks in this protected workspace are updated only when the artifact is published in the same workspace.

Updated endpoint:

• Update Workspace

A narrower yet more reasonable time range limit has been applied on the task execution filtering to optimize the efficiency of your execution history queries and prevent against misuse of the resources.

• When setting a time range with the from and the to parameters, the allowed maximum value is now 60 days.
• If no time range is explicitly set, the default value is the last 60 days.

• Get available task executions

The downloads of public API specifications are now bug free and ready for a smooth OAS (OpenAPI Specification) CLI generation.

Related endpoints:

• Orchestration
• Workspace Permissions
• Processing

You can now easily read who paused or resumed a task and on which date this operation was performed.

• Pause / Resume a Task

The endpoint to simulate runs of a given promotion has been relocated from the Processing domain to the Orchestration domain.

The old endpoint now becomes unavailable.

During this migration, some missing artifact version fields are also added in the response of the API call.

New endpoint:

• Perform promotion analysis

A number of discrepancy issues have been fixed for API endpoints to help you efficiently develop your API solutions.

Related endpoints:

• Get Plan details: the PlanPauseDetails attribute is present in the response for all the environments
• Create a schedule: use trigger instead of Trigger in the payload
• Update Task: version discrepancies have been removed in the response
• Create Task, Get available Tasks, and Update Task: in the three endpoints, the format inconsistency of the numeric values used in the context variables have been repaired.

All the project management endpoints, the Observability endpoints, and the smart timeout endpoints now support service account tokens. This means that you can use service accounts to invoke these APIs without having to rely on a user account.

• Projects
• Projects Authorization
• Observability Metrics

You can now use API requests to turn on or turn off smart timeout for a specific task.

For further information about how a smart timeout is computed, see Enabling smart Job task execution timeout.

These new endpoints support only the personal access tokens. They do not support the service account tokens.

You can now use the SCIM v2 API endpoints to automate the processes to create users, assign users to groups, assign roles to users and retrieve user profiles.

SCIM v2

These new endpoints are designed to help you easily synchronize role assignments and manage their changes between your SSO provider and Talend Cloud.

SSO Role Mapping

A new endpoint has been added to help you easily pause or resume plan executions, when required by a certain circumstance, for example, when a system maintenance window imposes a pause of plan executions.

Accordingly, related endpoints have been updated to take the plan pause status into account.

New endpoint:

• Pause-Resume Plan executions

Updated endpoints:

• Get available Plans executions
• Get Plan details

Maximum TPS (transaction Per Second) of API calls from an IP address has been set to 30 to ensure the resilience of the product for every API user.

This limit is meant to better handle peaks, protect from misuse or attacks, ensure equal capacities to all requests, and better adapt to situations from standard to specific.

For more limits about a Talend Cloud Management Console related acount, see Talend Cloud Management Console account limits.

A new endpoint has been created to simulate runs of a given promotion to build a detailed report accordingly.

This report helps you easily manage your software life cycle by better understanding the differences of artifacts and tasks between the source and the target environments of that promotion.

For the same purpose, the promotion execution endpoint is updated to include the artifact version information in its response.

New endpoint:

Updated endpoint:

• Execute a Promotion

In addition to the creation endpoints, more endpoints have been created to help you automate the management of workspaces and environments.

With these new endpoints, you can now set up automatic processes to perform the following operations:

• Environments:
  • get available environments
  • update a specific environment
  • delete an environment
• Workspaces:
  • get available workspaces
  • update a specific workspace
  • delete a workspace

Environment endpoints:

• List Environments
• Remove Environment
• Update Environment

Workspace endpoints:

• List Workspaces
• Remove Workspace
• Update Workspace

A new endpoint has been added to help you easily pause or resume task executions, when required by a certain circumstance, for example, when a system maintenance window imposes a pause of task executions.

Accordingly, related endpoints have been updated to take the task pause status into account.

New endpoint:

• Pause / Resume a Task

Updated endpoints:

• Get available Tasks
• Get Task by id
• Execute Task

In addition to the configuration level and the execution level, you can now set a timeout parameter in the trigger level of a Job task.

A timeout set on a trigger applies on the task run started by this trigger only. It overrides the task configuration timeout if that task has one.

Set this timeout when you append a trigger to a schedule or update a trigger of that schedule.

• Only time-based triggers support it; a webhook trigger does not.
• Its accuracy is 10 seconds.
• It is defined in minutes.
• It applies only on Job tasks running on Remote Engine Gen1 v2.12.0 and onwards, Remote Engine Cluster and Cloud Engine.

• Append some trigger to your schedule
• Update an existing trigger within the schedule

• Orchestration
• Processing
• Security
• Monitoring

These domains also contain a variety of subsets to bring the business oriented endpoints to an even finer granularity.

All these endpoints support both Personal Access Tokens (PATs) and Service Account Tokens (SATs)

See the following page to discover all these endpoints in their subsets and domains:

• APIs

• directly define multiple triggers for a task or plan. You do not need to duplicate your tasks or plans for each of those triggers anymore.

• simulate all the events to be triggered before applying a schedule to a task or plan
• manage changes in a schedule without taking any risk upon your current executions
• reuse and adapt existing schedules for other tasks or plans
• avoid redundant trigger definitions and tight event frequency, including the bissextile years "effect"
• protect against accidental deletion of a schedule when it is still used by a task or plan

Note that a schedule supports time triggers only. For this reason, a webhook trigger, which is merely event related, must be used as a separate option, that is to say, without being mixed with the time triggers.

For detailed use cases about how to use these endpoints with a task or a plan, see Using API to schedule task or plan runs

• Create a schedule
• Get all schedules
• Clone the schedule
• Update the schedule
• Retrieve the schedule
• Delete the schedule
• Update an existing trigger within the schedule
• Append some trigger to your schedule
• Remove a trigger from your schedule
• Simulate next triggered events
• Remove the association between a Task and a Schedule
• Associate a schedule to a Task.
• Retrieve task schedule
• Remove the association between the Schedule and the Plan
• Retrieve plan schedule
• Associate a schedule to a Plan.

• Search tasks

• Create Environment

• Create Workspace

You can set this parameter at two different levels to dynamically manage the timeout of your tasks:

• Task configuration level: set in the PUT request to the Configure Task executions endpoint. This timeout is added to the configuration of a task and thus applies on all the runs of this task.
• Task execution level: set in the POST request to the Execute Task endpoint. This timeout dynamically applies on the current task run only. It overrides the task configuration timeout if that task has one.

• Its accuracy is 10 seconds.
• It is defined in minutes.
• It applies only on Job tasks running on Remote Engine Gen1 v2.12.2 and onwards, Remote Engine Cluster and Cloud Engine.

• Configure Task executions
• Execute Task

• fileFormat: the format of the log file to be generated, JSON or TEXT.
• newFileName: this parameter indicates whether you need to include task IDs and execution IDs in the filename of the log to be generated. It is recommended to set this value to be true as the old filename format is going to be deprecated.

• Generate full logs for a completed or ongoing execution
• Get task execution logs

• Get task scheduled triggers
• Get plan scheduled triggers

• PromotionExecutableTaskV26
• PromotionExecutionStatusV26