Processing

Configure Engines and Clusters, manage and browse executions of your Tasks, Plans and Promotions.

Endpoints

Global security

These security schemes apply to the entire API

Security scheme

This scheme can be referenced across the API

Public
This Bearer authentication can be set in the Authorization header of your requests. Authentication tokens and Personal Access Tokens are supported. Personal Access Tokens can be generated in the Profile Preferences page of the Talend Cloud Portal.
Bearer authentication
Name Description
Format Bearer <token>

Task Executions

Executabletask

Object
Name Description Type Attributes and examples
executable Executable identifier string Required
Example
57f64991e4b0b689a64feed0
parameters Executable parameters ObjectOptional
logLevel Execution log level string Optional
Enum
  • OFF
  • ERROR
  • WARN
  • INFO
Example
WARN
timeout Execution timeout in minutes. Minimal value is 1. integer Optional
FormatINT32

JobExecutionStatus

Execution info
Object
Name Description Type Attributes and examples
executionId Job execution ID string Required
Example
7b2b122e-d6b8-42de-b0ba-fa2f0d36306e
startTimestamp Start time of job execution (UTC) datetime Required
FormatRFC3339
triggerTimestamp Trigger time of job execution (UTC) datetime Required
FormatRFC3339
finishTimestamp End time of job execution (UTC) datetime Optional
FormatRFC3339
userId User who triggered or scheduled the execution string Required
Example
fupton
userType Type of user who triggered or scheduled the execution string Required
Enum
  • HUMAN
  • SERVICE
jobId Job ID string Required
Example
57f64991e4b0b689a64feed0
jobVersion The version of the executed job string Required
Example
5.2
environmentVersion Execution environment version string Required
Enum
  • 1.2
  • 1.3
Example
1.3
executionStatus Execution status string Required
Enum
  • EXECUTION_EVENT_RECEIVED
  • DISPATCHING_FLOW
  • STARTING_FLOW_EXECUTION
  • STOPPING_FLOW_EXECUTION
  • EXECUTION_TERMINATED
  • DEPLOY_FAILED
  • EXECUTION_FAILED
  • EXECUTION_REJECTED
  • EXECUTION_SUCCESS
executionType Execution type string Required
Enum
  • MANUAL
  • SCHEDULED
  • WEBHOOK
  • PLAN
Example
SCHEDULED
executionDestination Destination of execution string Required
Enum
  • CLOUD
  • CLOUD_EXCLUSIVE
  • REMOTE_ENGINE
  • REMOTE_ENGINE_CLUSTER
Example
REMOTE_ENGINE
containerId Container ID string Optional
runProfileId Run profile ID string Optional
Example
157f818f-a901-4425-b592-0f9282687784
remoteEngineId Remote engine ID string Optional
Example
157f818f-a901-4425-b592-0f9282687784
remoteEngineClusterId Remote engine cluster ID string Optional
numberOfProcessedRows Number of processed rows integer Optional
FormatINT64
Example
1234567890
numberOfRejectedRows Number of rejected rows integer Optional
FormatINT64
Example
0
accountId Account ID string Required
Example
8494b016-b5ef-4b9c-b16d-8b1f824d7616
workspaceId Workspace ID string Required
Example
57ce63d3e4b0681c36d1a1c4
planId Plan execution ID string Optional
errorType Error type if error occurs string Optional
errorMessage Error message if error occurs string Optional
timeout Execution timeout if set (in minutes) integer Optional
FormatINT32

JobExecutionStatusV21

Execution info
Object
Name Description Type Attributes and examples
executionId Job execution ID string Required
Example
7b2b122e-d6b8-42de-b0ba-fa2f0d36306e
startTimestamp Start time of job execution (UTC) datetime Required
FormatRFC3339
triggerTimestamp Trigger time of job execution (UTC) datetime Required
FormatRFC3339
finishTimestamp End time of job execution (UTC) datetime Optional
FormatRFC3339
userId User who triggered or scheduled the execution string Required
Example
fupton
userType Type of user who triggered or scheduled the execution string Required
Enum
  • HUMAN
  • SERVICE
jobId Job ID string Required
Example
57f64991e4b0b689a64feed0
jobVersion The version of the executed job string Required
Example
5.2
environmentVersion Execution environment version string Required
Enum
  • 1.2
  • 1.3
Example
1.3
executionStatus Execution status string Required
Enum
  • EXECUTION_EVENT_RECEIVED
  • DISPATCHING_FLOW
  • STARTING_FLOW_EXECUTION
  • STOPPING_FLOW_EXECUTION
  • EXECUTION_TERMINATED
  • DEPLOY_FAILED
  • EXECUTION_FAILED
  • EXECUTION_REJECTED
  • EXECUTION_SUCCESS
  • EXECUTION_TERMINATED_TIMEOUT
executionType Execution type string Required
Enum
  • MANUAL
  • SCHEDULED
  • WEBHOOK
  • PLAN
Example
SCHEDULED
executionDestination Destination of execution string Required
Enum
  • CLOUD
  • CLOUD_EXCLUSIVE
  • REMOTE_ENGINE
  • REMOTE_ENGINE_CLUSTER
Example
REMOTE_ENGINE
containerId Container ID string Optional
runProfileId Run profile ID string Optional
Example
157f818f-a901-4425-b592-0f9282687784
remoteEngineId Remote engine ID string Optional
Example
157f818f-a901-4425-b592-0f9282687784
remoteEngineClusterId Remote engine cluster ID string Optional
numberOfProcessedRows Number of processed rows integer Optional
FormatINT64
Example
1234567890
numberOfRejectedRows Number of rejected rows integer Optional
FormatINT64
Example
0
accountId Account ID string Required
Example
8494b016-b5ef-4b9c-b16d-8b1f824d7616
workspaceId Workspace ID string Required
Example
57ce63d3e4b0681c36d1a1c4
planId Plan execution ID string Optional
errorType Error type if error occurs string Optional
errorMessage Error message if error occurs string Optional
status Task execution status string Required
Enum
  • dispatching
  • deploy_failed
  • executing
  • execution_rejected
  • execution_successful
  • execution_failed
  • terminated
  • terminated_timeout
Example
execution_successful

WorkspaceInfo

Object
Name Description Type Attributes and examples
id Workspace identifier string Optional
Example
57f64991e4b0b689a64feed2
name Workspace name string Optional
Example
Personal
description Workspace description string Optional
Example
workspace detail description
owner Workspace owner string Optional
Example
admin
type Workspace type string Optional
Enum
  • shared
  • personal
  • custom
Example
custom
environment EnvironmentInfo Optional
protectedArtifactUpdate

Allows to restrict the scope of task updates when a new artifact version is published. 

  • true : the tasks of this workspace are protected from the unintended updates triggered by artifacts being published in other workspaces. These tasks are updated only when artifacts are being published in the same workspace
  • false : tasks that have been set to use the latest version of the published artifact, or any other artifact that has the same name as the newly published artifact, are all updated, whatever workspace is used for this artifact publishing
 boolean Optional

TaskExecutionsFilters

Object
Name Description Type Attributes and examples
environmentId Environment filter string Optional
Example
6148511d6ccc6d41563574e5
workspaceId Workspace filter string Optional
Example
6148511d6ccc6d41563574e5
status Filter on status of execution : dispatching, deploy_failed, executing, execution_successful, execution_rejected, execution_failed, terminated, terminated_timeout string Optional
Enum
  • dispatching
  • deploy_failed
  • executing
  • execution_successful
  • execution_rejected
  • execution_failed
  • terminated
  • terminated_timeout
tags array of string Optional
Datatype details
Type Description Attributes and examples
array Task multiple tags filter
Example
critical + finance-eu
string One elementary tag to filter execution
Example
critical
from

Time period filter start date time (milliseconds).
With the from and to time period filters, the executions to be included in the response are:

  • Executions that started within the specified timeframe between from and to.
  • Executions that ended within the specified timeframe between from and to.
  • Executions that started and ended within the specified timeframe between from and to.
  • Executions that ran throughout the entire specified timeframe between from and to.
 number Optional
to

Time period filter end date time (milliseconds).
With the from and to time period filters, the executions to be included in the response are:

  • Executions that started within the specified timeframe between from and to.
  • Executions that ended within the specified timeframe between from and to.
  • Executions that started and ended within the specified timeframe between from and to.
  • Executions that ran throughout the entire specified timeframe between from and to.
 number Optional
limit Number of items to return. Must be in range [1, 100]. Default value - 100 number Optional
offset Position of the first item. Must be greater than or equal to 0. Default value - 0 number Optional
lastDays Number of days in the past number Optional
Example
1
TaskExecutionsFilters
{
  "environmentId": "57f64991e4b0b689a64feed3",
  "workspaceId": "57f64991e4b0b689a64feed2",
  "tags": [
    "critical",
    "supply-asia"
  ],
  "limit": 100,
  "offset": 10,
  "status": "execution_rejected",
  "from": 1646159400000,
  "to": 1649183400000,
  "lastDays": 5
}

Get available task executions

POST /processing/executables/tasks/executions

Get available task executions. Available filters are :

  • some specific environment as the alternate filter to workspaces
  • some specific workspace as the alternate filter to environments
  • status of execution
  • tags associated with tasks : when multiple tags are provided, each of them will be used as a filter, that is to say, the logical AND is applied
  • limit to a number of days in recent history (for example, 1 : yesterday, 5 : past five days)
  • datetime period in the past, not exceeding a 60-day range

All those parameters are optional and can be combined at your convenience. Note that the workspace should belong to the environment when both filters (environmentId and workspaceId) are specified in the same call.

When neither the number of days nor a datetime period is provided, the default 60 days time range is applied.

Use cases

Request

Body
application/json
TaskExecutionsFilters
application/json 
{
  "limit": 100,
  "offset": 10,
  "status": "executing",
  "from": 1646159400000,
  "to": 1649183400000,
  "lastDays": 20,
  "environmentId": "57f64991e4b0b689a64feed3",
  "workspaceId": "57f64991e4b0b689a64feed2",
  "tags": [
    "critical",
    "finance-eu"
  ]
}

Response

200Status 200
Ok
Body
application/json
Page
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get Tasks Executions

GET /processing/executables/tasks/executions

Get task executions. Available filters are :

  • some specific environment as the alternate filter to workspaces
  • some specific workspace as the alternate filter to environments
  • status of execution
  • limit to a number of days in recent history (for example, 1 : yesterday, 5 : past five days)
  • datetime period in the past, not exceeding a 60-day range

All those parameters are optional and can be combined at your convenience. Note that the workspace should belong to the environment when both filters (environmentId and workspaceId) are specified in the same call.

When neither the number of days nor a datetime period is provided, the default 60 days time range is applied.

Request

Query parameters
Name Description Type Attributes and examples
environmentId Environment filter string Optional
Example
6148511d6ccc6d41563574e1
workspaceId Workspace filter string Optional
Example
6148511d6ccc6d41563574e5
status Execution status string Optional
Enum
  • dispatching
  • deploy_failed
  • executing
  • execution_successful
  • execution_rejected
  • execution_failed
  • terminated
lastDays Number of days in the past integer Optional
FormatINT32
from Time period filter start date time (milliseconds) integer Optional
FormatINT64
to Time period filter end date time (milliseconds) integer Optional
FormatINT64
limit Number of items to return. Must be in range [1, 100]. Default value - 100 integer Optional
FormatINT32
offset Position of the first item. Must be greater than or equal to 0. Default value - 0 integer Optional
FormatINT32

Response

200Status 200
OK
Body
application/json
Page
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get Task Executions

GET /processing/executables/tasks/{taskId}/executions
Get task executions

Use cases

Request

Path variables
Name Description Type Attributes and examples
taskId task ID string Required
Query parameters
Name Description Type Attributes and examples
limit the number of items to return. Must be in range [1, 100]. Default value - 100 integer Optional
FormatINT32
status execution status string Optional
Enum
  • dispatching
  • deploy_failed
  • executing
  • execution_successful
  • execution_rejected
  • execution_failed
  • terminated
lastDays lastDays integer Optional
FormatINT32
from from date time (milliseconds) integer Optional
FormatINT64
offset the position of the first item. Must be greater than or equal to 0. Default value - 0 integer Optional
FormatINT32
to to date time (milliseconds) integer Optional
FormatINT64

Response

200Status 200
OK
Body
application/json
Page
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Execute Task

POST /processing/executions
Execute Task

Use cases

Request

Body
application/json
Executabletask

Response

201Status 201
Execution started
Body
application/json
Executionidentifier
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
application/json
ErrorResponse
404Status 404
Not found - resource not found
Body
application/json
ErrorResponse
405Status 405
Current Task is paused, it can’t be executed
500Status 500
Server error - something went wrong on server side
Body
application/json
ErrorResponse
501Status 501
Task cannot be deployed because it is not configured
Body
application/json
ErrorResponse

Get Task execution status

GET /processing/executions/{executionId}
Get Task execution status

Request

Path variables
Name Description Type Attributes and examples
executionId execution ID string Required

Response

200Status 200
Successful response
Body
application/json
JobExecutionStatusV21
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
application/json
ErrorResponse
404Status 404
Not found - resource not found
Body
application/json
ErrorResponse
429Status 429
Too many requests
Body
application/json
ErrorResponse
500Status 500
Server error - something went wrong on server side
Body
application/json
ErrorResponse

Terminate Task execution

DELETE /processing/executions/{executionId}
Terminate Task execution

Request

Path variables
Name Description Type Attributes and examples
executionId execution ID string Required

Response

200Status 200
Execution already finished
Body
text/plain
string
204Status 204
Execution stopped succesfully
400Status 400
Bad Request
Body
text/plain
ErrorResponse
401Status 401
Unauthorized
Body
text/plain
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
text/plain
ErrorResponse
404Status 404
Not found - resource not found
Body
text/plain
ErrorResponse
500Status 500
Server error - something went wrong on server side
Body
text/plain
ErrorResponse

Plan Executions

PlanExecutable

Object
Name Description Type Attributes and examples
executable Executable identifier string Required
Example
b91cf8b2-5dd1-4b18-915b-4c447cee5267
executionPlanId The plan execution identifier in case of re-execution string Optional
Example
0798b8d1-0e12-472f-be02-a0f04e792daa
stepId The step identifier to start with string Optional
Example
09043c9f-02d0-41f6-b3cb-0ea53ffde377
rerunOnlyFailedTasks Optionally run only failing tasks boolean Optional

PlanExecutionStatus

Plan Execution info
Object
Name Description Type Attributes and examples
executionId Job execution ID string Required
Example
7b2b122e-d6b8-42de-b0ba-fa2f0d36306e
startTimestamp Start time of job execution (UTC) datetime Required
FormatRFC3339
finishTimestamp End time of job execution (UTC) datetime Optional
FormatRFC3339
userId User who triggered or scheduled the execution string Required
Example
fupton
userType Type of user who triggered or scheduled the execution string Required
Enum
  • HUMAN
  • SERVICE
planId Plan ID string Required
Example
b91cf8b2-5dd1-4b18-915b-4c447cee5267
executionStatus Execution (detailed) status string Required
Enum
  • UNDEFINED
  • STARTED
  • FINISHED
  • EXECUTION_FAILED
  • EXECUTION_SUCCESS
Example
FINISHED
plannedExecutableCount Number of planned executables integer Optional
FormatINT32
Example
2
doneExecutableCount Number of done executables integer Optional
FormatINT32
Example
1
doneExecutableDetails array of JobExecutionStatus Optional
Datatype details
Type Description Attributes and examples
array Execution activity info
JobExecutionStatus

StepExecution

Object
Name Description Type Attributes and examples
id string Optional
executionId string Optional
startTimestamp datetime Optional
FormatRFC3339
finishTimestamp datetime Optional
FormatRFC3339
executionStatus string Optional

Execute Plan

POST /processing/executions/plans

Allows to run a Plan.

This endpoint also allows for re-running some plan execution from some failed step : you should then provide execution id and step from which you wish to restart execution.

Request

Body
application/json
PlanExecutable

Response

201Status 201
Execution started
Body
application/json
Executionidentifier
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
application/json
ErrorResponse
404Status 404
Not found - resource not found
Body
application/json
ErrorResponse
405Status 405
Plan is paused, it can’t be executed
422Status 422
Entity exists but cannot be processed, because too old or plan exec didn’t fail
Body
application/json
ErrorResponse
500Status 500
Server error - something went wrong on server side
Body
application/json
ErrorResponse
501Status 501
Plan cannot be deployed because it is not configured
Body
application/json
ErrorResponse

Get available Plans executions

GET /processing/executables/plans/executions
List all plan executions the user is granted visibility on. For each plan execution returns execution and plan identifiers, start and finish timestamps, execution status and counters on done and planned executables within plan. This query is paginated and accepts filters as defined in parameters documented above.

Request

Query parameters
Name Description Type Attributes and examples
workspaceId workspace ID the plans should belong to string Optional
limit the number of items to return. Must be in range [1, 100]. Default value - 100 integer Optional
FormatINT32
status execution status string Optional
Enum
  • executing
  • execution_successful
  • execution_failed
lastDays lastDays integer Optional
FormatINT32
from from date time (milliseconds) integer Optional
FormatINT64
offset the position of the first item. Must be greater than or equal to 0. Default value - 0 integer Optional
FormatINT32
environmentId environment ID the plans should belong to string Optional
to to date time (milliseconds) integer Optional
FormatINT64

Response

200Status 200
OK
Body
application/json
Page
Status 200 application/json 
{
  "items": [
    {
      "executionId": "7fb48c7a-5ea5-4cb9-9333-e345f28d9dd5",
      "startTimestamp": "2022-10-20T14:20:28.582Z",
      "finishTimestamp": "2022-10-20T14:20:39.830Z",
      "planId": "e7734729-cf3d-4578-939c-30381e95cd10",
      "executionStatus": "EXECUTION_SUCCESS",
      "plannedExecutableCount": 1,
      "doneExecutableCount": 1,
      "status": "execution_successful"
    },
    {
      "executionId": "dc57c82f-7f18-4228-9f82-a81e02bae617",
      "startTimestamp": "2022-10-20T14:17:56.390Z",
      "finishTimestamp": "2022-10-20T14:18:07.865Z",
      "planId": "e7734729-cf3d-4578-939c-30381e95cd10",
      "executionStatus": "EXECUTION_SUCCESS",
      "plannedExecutableCount": 1,
      "doneExecutableCount": 1,
      "status": "execution_successful"
    },
    {
      "executionId": "4555c45f-9548-41e9-98b4-a9bfa2ee4e5c",
      "startTimestamp": "2022-10-20T13:47:37.611Z",
      "finishTimestamp": "2022-10-20T13:47:48.864Z",
      "planId": "e7734729-cf3d-4578-939c-30381e95cd10",
      "executionStatus": "EXECUTION_SUCCESS",
      "plannedExecutableCount": 1,
      "doneExecutableCount": 1,
      "status": "execution_successful"
    },
    {
      "executionId": "f3cfb077-2ea6-4557-bbb2-fd027f3cae4e",
      "startTimestamp": "2022-10-20T13:45:04.027Z",
      "finishTimestamp": "2022-10-20T13:45:15.929Z",
      "planId": "e7734729-cf3d-4578-939c-30381e95cd10",
      "executionStatus": "EXECUTION_SUCCESS",
      "plannedExecutableCount": 1,
      "doneExecutableCount": 1,
      "status": "execution_successful"
    },
    {
      "executionId": "b668cd50-6421-4e57-b15a-79b39ac153a8",
      "startTimestamp": "2022-10-20T13:18:01.235Z",
      "finishTimestamp": "2022-10-20T13:18:16.093Z",
      "planId": "e7734729-cf3d-4578-939c-30381e95cd10",
      "executionStatus": "EXECUTION_FAILED",
      "plannedExecutableCount": 1,
      "doneExecutableCount": 1,
      "status": "execution_failed"
    },
    {
      "executionId": "226dc12c-a288-4fe0-96f6-f6d51fa2b9ff",
      "startTimestamp": "2022-10-20T12:08:48.823Z",
      "finishTimestamp": "2022-10-20T12:18:52.729Z",
      "planId": "e7734729-cf3d-4578-939c-30381e95cd10",
      "executionStatus": "EXECUTION_SUCCESS",
      "plannedExecutableCount": 1,
      "doneExecutableCount": 1,
      "status": "execution_successful"
    }
  ],
  "limit": 100,
  "offset": 0,
  "total": 6
}
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get all steps for a plan execution (order by designed execution, only steps, without error handlers)

GET /processing/executions/plans/{planExecutionId}/steps
Get Steps executions status

Request

Path variables
Name Description Type Attributes and examples
planExecutionId Plan execution ID string Required

Response

200Status 200
Successful response
Body
application/json
details
Name Description Type Attributes and examples
array of StepExecution
Datatype details
Type Description Attributes and examples
array
StepExecution
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
application/json
ErrorResponse
404Status 404
Not found - resource not found
Body
application/json
ErrorResponse
500Status 500
Server error - something went wrong on server side
Body
application/json
ErrorResponse

Get Plan executions

GET /processing/executables/plans/{planId}/executions
Get Plan executions

Request

Path variables
Name Description Type Attributes and examples
planId plan ID string Required
Query parameters
Name Description Type Attributes and examples
limit the number of items to return. Must be in range [1, 100]. Default value - 100 integer Optional
FormatINT32
status execution status string Optional
Enum
  • executing
  • execution_successful
  • execution_failed
lastDays lastDays integer Optional
FormatINT32
from from date time (milliseconds) integer Optional
FormatINT64
offset the position of the first item. Must be greater than or equal to 0. Default value - 0 integer Optional
FormatINT32
to to date time (milliseconds) integer Optional
FormatINT64

Response

200Status 200
OK
Body
application/json
Page
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get Plan Execution status

GET /processing/executions/plans/{id}
Get detailed status of one plan execution

Request

Path variables
Name Description Type Attributes and examples
id Execution id string Required

Response

200Status 200
Detailed status on one Platform execution
Body
application/json
PlanExecutionStatus
400Status 400
Status 400
Body
*/*
ErrorResponse
401Status 401
Status 401
Body
application/json
ErrorResponse
403Status 403
Permissions required
Body
*/*
ErrorResponse
404Status 404
Execution not found
Body
*/*
ErrorResponse
429Status 429
Status 429
Body
*/*
ErrorResponse
500Status 500
Something went wrong on server side
Body
*/*
ErrorResponse

Get the step for plan execution without error handlers

GET /processing/executions/plans/{planExecutionId}/steps/{stepExecutionId}
Get Step execution status

Request

Path variables
Name Description Type Attributes and examples
planExecutionId Plan execution ID string Required
stepExecutionId Step execution ID string Required

Response

200Status 200
Successful response
Body
application/json
PlanExecutionStatus
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
application/json
ErrorResponse
404Status 404
Not found - resource not found
Body
application/json
ErrorResponse
500Status 500
Server error - something went wrong on server side
Body
application/json
ErrorResponse

Get execution status of the handler for the specified step

GET /processing/executions/plans/{planExecutionId}/steps/{stepExecutionId}/error-handler
Get Step handler execution status

Request

Path variables
Name Description Type Attributes and examples
planExecutionId Plan execution ID string Required
stepExecutionId Step execution ID string Required

Response

200Status 200
Successful response
Body
application/json
PlanExecutionStatus
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden - no permissions to access resource
Body
application/json
ErrorResponse
404Status 404
Not found - resource not found
Body
application/json
ErrorResponse
500Status 500
Server error - something went wrong on server side
Body
application/json
ErrorResponse

Promotion Executions

AdvancedPromotionSpec

Object
Name Description Type Attributes and examples
artifactId Advanced Promotion Artifact Id string Optional
Example
57f64991e4b0b689a64feed0
artifactType Advanced Promotion Artifact Type. In the following list of allowed values, ‘ACTION’ refers to Artifacts, ‘FLOW’ refers to Tasks, ‘PLAN’ refers to Plans and ‘WORKSPACE’ refers to Workspaces string Optional
Enum
  • WORKSPACE
  • PLAN
  • FLOW
  • ACTION
Example
WORKSPACE

ArtifactPromotionResult

Object
Name Description Type Attributes and examples
id Artifact ID string Optional
targetId Target Artifact ID string Optional
name Artifact Name string Optional
type Artifact Type string Optional
Enum
  • WORKSPACE
  • PLAN
  • FLOW
  • ACTION
  • CONNECTION
  • RESOURCE
  • ENGINE
  • CLUSTER
  • FOLDER
Example
WORKSPACE
analyzeReport Report Optional
promotionReport Report Optional
usedBy array of PromotionResultInfo Optional
Datatype details
Type Description Attributes and examples
array Used by Artifacts
PromotionResultInfo
targetVersion Artifact Version In Target Environment string Optional
Example
"0.1.1.20223008022926"
version Artifact Version In Source Environment string Optional
Example
"0.1.1.20223008022926"
artifactId Flow artifact Id string Optional
Example
DataManagementJob_1
jobType Action Job Type string Optional
Example
STANDARD
versions array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array List of Action Versions
ArtifactPromotionResult

Execution

Object
Name Description Type Attributes and examples
executionId string Optional
executionReport PromotionExecutionStatusV26 Optional

FlowPromotionResult

Object
Name Description Type Attributes and examples
id Artifact ID string Optional
targetId Target Artifact ID string Optional
name Artifact Name string Optional
type Artifact Type string Optional
Example
FLOW
analyzeReport Promotion Analysis Stage Report Report Optional
promotionReport Promotion Execution Stage Report Report Optional
usedBy Actions that use this flow FlowPromotionResult Optional
targetVersion Promoted Flow Version In Target Environment string Optional
Example
"1.1"
version Source Flow Version integer Optional
FormatINT32
Example
1
versions array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Action Versions
ArtifactPromotionResult
jobType Flow Job Type string Optional
Example
STANDARD
workspaceId Promoted Flow Workspace ID string Optional
artifactId Flow Artifact ID string Optional
Example
Data_management_job_1

PromotionExecutableTaskV26

Object
Name Description Type Attributes and examples
executable Executable identifier string Required
Example
57f64991e4b0b689a64feed0
keepTargetResources Allows to keep target resources untouched. boolean Optional
keepTargetRunProfiles Allows to keep target run profiles untouched. boolean Optional
advanced Advance option to narrow the scope of the promotion execution : for instance limit to a workspace, etc. When not provided, promotion will apply on the whole environment scope. AdvancedPromotionSpec Optional
context Execution context message string Optional
Max length255
Example
Free text describing the context for promotion execution

PromotionExecutionStatus

Object
Name Description Type Attributes and examples
executionId Job execution ID string Required
Example
7b2b122e-d6b8-42de-b0ba-fa2f0d36306e
startTimestamp Start time of job execution (UTC) datetime Required
FormatRFC3339
triggerTimestamp Trigger time of job execution (UTC) datetime Required
FormatRFC3339
finishTimestamp End time of job execution (UTC) datetime Optional
FormatRFC3339
userId User who triggered or scheduled the execution string Required
Example
fupton
promotionId Promotion ID string Optional
pipelineId Deprecated! Use Promotion ID instead string Optional
keepTargetResources Keep Target Resources flag boolean Optional
keepTargetRunProfiles Keep Target Run Profiles flag (not returned for API versions earlier V2.2) boolean Optional
advanced AdvancedPromotionSpec Optional
defective Defective Promotion flag boolean Optional
status Execution status string Optional
Enum
  • ANALIZED
  • PROMOTION_STARTED
  • PROMOTION_WARNING
  • PROMOTION_FAILED
  • PROMOTED
Example
PROMOTED
statusMessage Execution status message string Optional
workspaces array of WorkspacePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Workspace Promotion Execution Info
WorkspacePromotionResult
engines array of RuntimePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Remote Engines Promotion Execution Info
RuntimePromotionResult
clusters array of RuntimePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Remote Engine Clusters Promotion Execution Info
RuntimePromotionResult

PromotionExecutionStatusV26

Object
Name Description Type Attributes and examples
executionId Job execution ID string Required
Example
7b2b122e-d6b8-42de-b0ba-fa2f0d36306e
startTimestamp Start time of job execution (UTC) datetime Required
FormatRFC3339
triggerTimestamp Trigger time of job execution (UTC) datetime Required
FormatRFC3339
finishTimestamp End time of job execution (UTC) datetime Optional
FormatRFC3339
userId User who triggered or scheduled the execution string Required
Example
fupton
promotionId Promotion ID string Optional
pipelineId Deprecated! Use Promotion ID instead string Optional
keepTargetResources Keep Target Resources flag boolean Optional
keepTargetRunProfiles Keep Target Run Profiles flag (not returned for API versions earlier V2.2) boolean Optional
advanced AdvancedPromotionSpec Optional
defective Defective Promotion flag boolean Optional
status Execution status string Optional
Enum
  • ANALIZED
  • PROMOTION_STARTED
  • PROMOTION_WARNING
  • PROMOTION_FAILED
  • PROMOTED
Example
PROMOTED
statusMessage Execution status message string Optional
workspaces array of WorkspacePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Workspace Promotion Execution Info
WorkspacePromotionResult
engines array of RuntimePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Remote Engines Promotion Execution Info
RuntimePromotionResult
clusters array of RuntimePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Remote Engine Clusters Promotion Execution Info
RuntimePromotionResult
context Execution context message string Optional
Max length255
Example
Free text describing the context for promotion execution
userType Promotion executor type, either HUMAN or SERVICE string Optional
Example
SERVICE

PromotionResultInfo

Object
Name Description Type Attributes and examples
id Artifact ID string Optional
targetId Target Artifact ID string Optional
name Artifact Name string Optional
type Artifact Type string Optional
Enum
  • WORKSPACE
  • PLAN
  • FLOW
  • ACTION
  • CONNECTION
  • RESOURCE
  • ENGINE
  • CLUSTER
  • FOLDER
Example
WORKSPACE
analyzeReport Report Optional
promotionReport Report Optional
targetVersion Applicable to flows only, displays version of flow in target environment after promotion execution string Optional
Example
1.1
version Applicable to flows only, displays version of flow in source environment integer Optional
FormatINT32

Report

Object
Name Description Type Attributes and examples
status string Optional
Enum
  • CREATED
  • REPLACED
  • REUSED
  • FAILED
  • SKIPPED
messages array of Message Optional
Datatype details
Type Description Attributes and examples
array
Message

RuntimePromotionResult

Object
Name Description Type Attributes and examples
id Artifact ID string Optional
targetId Target Artifact ID string Optional
name Artifact Name string Optional
type Artifact Type string Optional
Enum
  • WORKSPACE
  • PLAN
  • FLOW
  • ACTION
  • CONNECTION
  • RESOURCE
  • ENGINE
  • CLUSTER
  • FOLDER
Example
WORKSPACE
analyzeReport Report Optional
promotionReport Report Optional
usedBy array of PromotionResultInfo Optional
Datatype details
Type Description Attributes and examples
array Used by Artifacts
PromotionResultInfo
runProfiles array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Run profiles
ArtifactPromotionResult

WorkspacePromotionResult

Object
Name Description Type Attributes and examples
id Artifact ID string Optional
targetId Target Artifact ID string Optional
name Artifact Name string Optional
type Artifact Type string Optional
Enum
  • WORKSPACE
  • PLAN
  • FLOW
  • ACTION
  • CONNECTION
  • RESOURCE
  • ENGINE
  • CLUSTER
  • FOLDER
Example
WORKSPACE
workspaceType Promoted workspace type, can be either Shared or Custom string Optional
Example
Shared
analyzeReport Report Optional
promotionReport Report Optional
flows array of FlowPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Flows Promotion Execution Info
FlowPromotionResult
plans array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Plans Promotion Execution Info
ArtifactPromotionResult
actions array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Actions Promotion Execution Info
ArtifactPromotionResult
connections array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Connections Promotion Execution Info
ArtifactPromotionResult
resources array of ArtifactPromotionResult Optional
Datatype details
Type Description Attributes and examples
array Resources Promotion Execution Info
ArtifactPromotionResult
engines array of RuntimePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Engines Promotion Execution Info
RuntimePromotionResult
clusters array of RuntimePromotionResult Optional
Datatype details
Type Description Attributes and examples
array Clusters Promotion Execution Info
RuntimePromotionResult

Execute a Promotion

POST /processing/executions/promotions

Execute a Promotion from source Environment to target one, according to all the default configuration set on Tasks.

Detailed rules on promotion executions are available here : https://help.talend.com/r/en-US/Cloud/management-console-user-guide/promotion-rules

In addition to these rules, current execution can be adapted as such :

  • Default scope of Promotion is the whole environment. When narrower scope (Workspace, Task, etc.) please provide advanced option parameter accordingly.
  • Run Profiles and Resources are defaulty updated on target. You can also choose to keep them untouched using parameters.

Lastly, promotion simulation and analysis is also available: helpful when check before exec is needed. Automatic-undo of a promotion execution is not provided.

Request

Body
application/json
PromotionExecutableTaskV26

Response

201Status 201
Execution success
Body
application/json
Execution
202Status 202
Execution started
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse
501Status 501
Promotions can’t be executed because it has errors
Body
application/json
ErrorResponse

Get Promotion execution status

GET /processing/executions/promotions/{executionId}
Get Promotion execution status

Request

Path variables
Name Description Type Attributes and examples
executionId Execution ID string Required

Response

200Status 200
OK
Body
application/json
PromotionExecutionStatusV26
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Remote Engines

Engine

Object
Name Description Type Attributes and examples
id Engine id string Required
Example
5cdd2ce2c737973f9d581b98
name Engine name string Required
Example
Remote Engine
description Engine description string Optional
Example
Some description
workspace WorkspaceInfo Optional
createDate Date of creation of the engine datetime Required
FormatRFC3339
updateDate Date of updating of the engine datetime Optional
FormatRFC3339
runtimeId Engine runtime id string Required
Example
9e8acf72-00d5-451b-9175-97bd44cd6b13
availability Availability status of engine|cluster string Optional
Enum
  • AVAILABLE
  • NOT_AVAILABLE
  • RETIRED
Example
NOT_AVAILABLE
managed Indicates whether target runtime (engine/cluster) is managed or not boolean Optional
status Engine status string Required
Enum
  • PAIRED
  • NOT_PAIRED
Example
NOT_PAIRED
runProfiles array of string Required
Datatype details
Type Description Attributes and examples
array Run profiles of engine
Example
["MICROSERVICE"]
string
debug EngineDebug Optional
clusterId cluster that this engine is part of string Optional
preAuthorizedKey Preauthorized key (only for not paired engine) string Optional
Example
123A822BF26DF49045C90D94CD269F3850BFB308B2606019CC63EDADE55B4391
esbCompatibilityVersion ESB compatibility version string Optional
Example
7.3.1

EngineDebug

Object
Name Description Type Attributes and examples
host Debug host of engine string Required
Example
192.168.0.1

EngineRequest

Object
Name Description Type Attributes and examples
name Engine name string Required
Example
Remote Engine
environmentId Id of engine environment string Required
Example
5c1111d7a4186a4eafed0587
workspaceId Id of engine workspace string Optional
Example
5c2222d7a4186a4eafed0587
runProfiles array of string Optional
Datatype details
Type Description Attributes and examples
array Run profiles of engine
Example
["MICROSERVICE"]
string
debug EngineDebug Optional
description Engine|cluster description string Optional
Example
Some description
esbCompatibilityVersion Data Service Runner/Talend Runtime version. Mandatory if TALEND_RUNTIME or MICROSERVICE profile selected string Optional
Example
7.2.1

RunProfileRequest

Object
Name Description Type Attributes and examples
name Run profile name string Required
Example
Run profile
description Description string Optional
Example
Run profile with default options
jvmArguments array of string Required
Datatype details
Type Description Attributes and examples
array Jvm arguments
Example
["-Xms128m","-Xmx256m"]
string
type Run profile type string Required
Enum
  • JOB_SERVER
  • MICROSERVICE
  • TALEND_RUNTIME
Example
JOB_SERVER
runtimeId Runtime id string Required
Example
5e713618de3c75a450447262

RunProfileUpdateRequest

Object
Name Description Type Attributes and examples
name Run profile name string Required
Example
Run profile
description Description string Optional
Example
Run profile with default options
jvmArguments array of string Required
Datatype details
Type Description Attributes and examples
array Jvm arguments
Example
["-Xms128m","-Xmx256m"]
string

RuntimeRunProfile

Object
Name Description Type Attributes and examples
id string Optional
name string Optional
description string Optional
createDate datetime Optional
FormatRFC3339
updateDate datetime Optional
FormatRFC3339
type Run profile type string Optional
Enum
  • JOB_SERVER
  • MICROSERVICE
  • TALEND_RUNTIME
jvmArguments array of string Optional
Datatype details
Type Description Attributes and examples
array Jvm arguments
string
runtimeId string Optional
version Run profile version integer Optional
FormatINT32

Create new Remote Engine

POST /processing/runtimes/remote-engines
Create new Remote Engine

Request

Body
application/json
EngineRequest

Response

201Status 201
Created
Body
application/json
Engine
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get all (available) Remote Engines

GET /processing/runtimes/remote-engines
Get all (available) Remote Engines

Request

Query parameters
Name Description Type Attributes and examples
query search query (FIQL format) string Optional
Example
workspace.environment.id==5cb47ca4b1b5247f6006529e

Response

200Status 200
OK
Body
application/json
details
Name Description Type Attributes and examples
array of Engine
Datatype details
Type Description Attributes and examples
array
Engine
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Delete Remote Engine by id

DELETE /processing/runtimes/remote-engines/{id}
Delete Remote Engine by id

Request

Path variables
Name Description Type Attributes and examples
id remote engine id string Required

Response

204Status 204
No Content
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
409Status 409
Conflict
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get Remote Engine by id

GET /processing/runtimes/remote-engines/{id}
Get Remote Engine by id

Request

Path variables
Name Description Type Attributes and examples
id remote engine id string Required

Response

200Status 200
OK
Body
application/json
Engine
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Unpair Remote Engine

DELETE /processing/runtimes/remote-engines/{id}/pairing

Unpair Remote Engine.

If data service tasks and Route tasks are available in your license, you can maintain OSGi Data Services as deployed while upgrading your Remote Engine, without need to undeploy and redeploy those services. See Upgrading Remote Engine v2.11.8 onwards for data service Job or Route tasks for more details about this specific upgrade process.

Request

Path variables
Name Description Type Attributes and examples
id remote engine id string Required
Query parameters
Name Description Type Attributes and examples
keepOsgiDeployments Boolean to indicate if OSGi Routes or data services should be kept as deployed while unpairing - used for Remote Engine upgrade with data service tasks and Route tasks only boolean Optional
Example
https://api.us.cloud.talend.com/processing/runtimes/remote-engines/640609d2092883085a6eb9e2/pairing?keepOsgiDeployments=true
keepMicroserviceDeployments Boolean to indicate if microservice Routes or data services should be kept as deployed while unpairing - used for Remote Engine upgrade with data service tasks and Route tasks only boolean Optional
Example
https://api.us.cloud.talend.com/processing/runtimes/remote-engines/640609d2092883085a6eb9e2/pairing?keepMicroserviceDeployments=true

Response

200Status 200
OK
Body
text/plain
string
400Status 400
Bad Request
Body
text/plain
ErrorResponse
401Status 401
Unauthorized
Body
text/plain
ErrorResponse
403Status 403
Forbidden
Body
text/plain
ErrorResponse
404Status 404
Not Found
Body
text/plain
ErrorResponse
500Status 500
Internal Server Error
Body
text/plain
ErrorResponse

Create engine run profile

POST /processing/runtimes/remote-engines/{engineId}/run-profiles
Create engine run profile

Request

Path variables
Name Description Type Attributes and examples
engineId remote engine id string Required
Body
application/json
RunProfileRequest

Response

201Status 201
Created
Body
application/json
RuntimeRunProfile
400Status 400
Bad run profile definition
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
No engine can be found by provided
Body
application/json
ErrorResponse
409Status 409
Run profile with corresponding name already exists for given runtime
500Status 500
Internal server error
Body
application/json
ErrorResponse

Get all engine run profiles

GET /processing/runtimes/remote-engines/{engineId}/run-profiles
Get all engine run profiles

Request

Path variables
Name Description Type Attributes and examples
engineId remote engine id string Required

Response

200Status 200
OK
Body
application/json
details
Name Description Type Attributes and examples
array of RuntimeRunProfile
Datatype details
Type Description Attributes and examples
array
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Engine not found
Body
application/json
ErrorResponse

Get engine run profile by id

GET /processing/runtimes/remote-engines/{engineId}/run-profiles/{runProfileId}
Get engine run profile by id

Request

Path variables
Name Description Type Attributes and examples
engineId remote engine id string Required
runProfileId run profile id string Required

Response

200Status 200
OK
Body
application/json
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Run profile not found
Body
application/json
ErrorResponse

Update engine run profile

PUT /processing/runtimes/remote-engines/{engineId}/run-profiles/{runProfileId}
Update engine run profile

Request

Path variables
Name Description Type Attributes and examples
engineId remote engine id string Required
runProfileId run profile id string Required
Body
application/json
RunProfileUpdateRequest

Response

200Status 200
OK
Body
application/json
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Run profile not found
Body
application/json
ErrorResponse
500Status 500
Internal server error
Body
application/json
ErrorResponse

Delete engine run profile

DELETE /processing/runtimes/remote-engines/{engineId}/run-profiles/{runProfileId}

Delete engine run profile.

When in use in some task configuration, deletion is aborted unless you provided some new Run Profile id to use. It that latter case, tasks will be using this other Run Profile and the original Run Profile would be deleted.

Request

Path variables
Name Description Type Attributes and examples
engineId remote engine id string Required
runProfileId run profile id string Required
Query parameters
Name Description Type Attributes and examples
newRunProfileId new run profile id to use in tasks string Optional

Response

202Status 202
Run Profile deleted.
403Status 403
Forbidden
Body
*/*
ErrorResponse
404Status 404
No run profile can be found
Body
*/*
ErrorResponse
409Status 409
Run Profile is in use in some task configuration
Body
application/json
List of Task ID that are using the Run Profile.
string
500Status 500
Internal server error
Body
*/*
ErrorResponse

Remote Engine Clusters

ClusterRequest

Object
Name Description Type Attributes and examples
name Cluster name string Required
Example
Some cluster name
workspaceId Workspace identifier string Required
Example
5c935c8ba4186a4eafed0592
environmentId Environment identifier string Required
Example
5c935c8ba4186a4eafed0592
description Cluster description string Required
Example
some description
remoteEngines array of string Required
Datatype details
Type Description Attributes and examples
array List of remote engine ids
Example
["5d3039de65ff7d3583840bc0","5d3039de65ff7d3583840bk1"]
string

EngineCluster

Object
Name Description Type Attributes and examples
id Resource id string Required
Example
5cdd2ce2c737973f9d581b98
name Resource name string Required
Example
Remote Engine
description Resource description string Optional
Example
Some description
workspace WorkspaceInfo Optional
createDate Date of creation of the resource datetime Required
FormatRFC3339
updateDate Date of updating of the resource datetime Optional
FormatRFC3339
runtimeId Resource runtime id string Required
Example
9e8acf72-00d5-451b-9175-97bd44cd6b13
availability Availability status of engine|cluster string Optional
Enum
  • AVAILABLE
  • NOT_AVAILABLE
  • RETIRED
Example
NOT_AVAILABLE
managed Indicates whether target runtime (engine/cluster) is managed or not boolean Optional
engines array of string Optional
Datatype details
Type Description Attributes and examples
array Engines in cluster
Example
["5d3039de65ff7d3583840bc0","5d3039de65ff7d3583840bk1"]
string

Create Remote Engine Cluster

POST /processing/runtimes/remote-engine-clusters
Create Remote Engine Cluster

Request

Body
application/json
ClusterRequest

Response

201Status 201
Created
Body
application/json
EngineCluster
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get all (available) Remote Engine Clusters

GET /processing/runtimes/remote-engine-clusters
Get all (available) Remote Engine Clusters

Request

Query parameters
Name Description Type Attributes and examples
_s search query (FIQL format), e.g. “workspace.environment.id==5cb47ca4b1b5247f6006529e”,“name==NewCluster” string Optional

Response

200Status 200
OK
Body
application/json
details
Name Description Type Attributes and examples
array of BaseEngine
Datatype details
Type Description Attributes and examples
array
BaseEngine
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get Remote Engine Cluster by id

GET /processing/runtimes/remote-engine-clusters/{clusterId}
Get Remote Engine Cluster by id

Request

Path variables
Name Description Type Attributes and examples
clusterId remote engine cluster id string Required

Response

200Status 200
OK
Body
application/json
EngineCluster
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Delete Remote Engine Cluster by id

DELETE /processing/runtimes/remote-engine-clusters/{clusterId}
Delete Remote Engine Cluster by id

Request

Path variables
Name Description Type Attributes and examples
clusterId remote engine cluster id string Required

Response

204Status 204
No Content
400Status 400
Bad Request
Body
*/*
ErrorResponse
401Status 401
Unauthorized
Body
*/*
ErrorResponse
403Status 403
Forbidden
Body
*/*
ErrorResponse
404Status 404
Not Found
Body
*/*
ErrorResponse
500Status 500
Internal Server Error
Body
*/*
ErrorResponse

Add Remote Engine to Remote Engine Cluster

PUT /processing/runtimes/remote-engine-clusters/{clusterId}/engines/{engineId}
Add Remote Engine to Remote Engine Cluster

Request

Path variables
Name Description Type Attributes and examples
clusterId remote engine cluster id string Required
engineId remote engine id string Required
Body
application/json
Optional reschedule option for the run profile
RunProfileRescheduleOptions

Response

204Status 204
No Content
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
409Status 409
Conflict
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Remove Remote Engine from Remote Engine Cluster

DELETE /processing/runtimes/remote-engine-clusters/{clusterId}/engines/{engineId}
Remove Remote Engine from Remote Engine Cluster

Request

Path variables
Name Description Type Attributes and examples
clusterId remote engine cluster id string Required
engineId remote engine id string Required

Response

204Status 204
No Content
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get cluster run profiles

GET /processing/runtimes/remote-engine-clusters/{clusterId}/run-profiles
Get cluster run profiles

Request

Path variables
Name Description Type Attributes and examples
clusterId cluster id string Required

Response

200Status 200
OK
Body
application/json
details
Name Description Type Attributes and examples
array of RuntimeRunProfile
Datatype details
Type Description Attributes and examples
array
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Cluster not found
Body
application/json
ErrorResponse

Create cluster run profile

POST /processing/runtimes/remote-engine-clusters/{clusterId}/run-profiles
Create cluster run profile

Request

Path variables
Name Description Type Attributes and examples
clusterId cluster id string Required
Body
application/json
RunProfileRequest

Response

201Status 201
400Status 400
Bad run profile definition
Body
application/json
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
No cluster can be found by provided
Body
application/json
ErrorResponse
409Status 409
Run profile name already used
500Status 500
Internal server error
Body
application/json
ErrorResponse

Update cluster run profile

PUT /processing/runtimes/remote-engine-clusters/{clusterId}/run-profiles/{runProfileId}
Update cluster run profile

Request

Path variables
Name Description Type Attributes and examples
clusterId cluster id string Required
runProfileId run profile id string Required
Body
application/json
RunProfileUpdateRequest

Response

200Status 200
OK
Body
application/json
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
No run profile can be found
Body
application/json
ErrorResponse
500Status 500
Internal server error
Body
application/json
ErrorResponse

Delete cluster run profile

DELETE /processing/runtimes/remote-engine-clusters/{clusterId}/run-profiles/{runProfileId}
Delete cluster run profile

Request

Path variables
Name Description Type Attributes and examples
clusterId cluster id string Required
runProfileId run profile id string Required
Query parameters
Name Description Type Attributes and examples
runProfileId new run profile id string Optional

Response

204Status 204
No Content
403Status 403
Forbidden
Body
*/*
ErrorResponse
404Status 404
No run profile can be found
Body
*/*
ErrorResponse
500Status 500
Internal server error
Body
*/*
ErrorResponse

Get cluster run profile

GET /processing/runtimes/remote-engine-clusters/{clusterId}/run-profiles/{runProfileId}
Get cluster run profile

Request

Path variables
Name Description Type Attributes and examples
clusterId cluster id string Required
runProfileId run profile id string Required

Response

200Status 200
OK
Body
application/json
RuntimeRunProfile
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Run profle not found
Body
application/json
ErrorResponse

RunProfileRescheduleOptions

Object
Name Description Type Attributes and examples
method Run profile reschedule method string Required
DefaultDEFAULT
Enum
  • DEFAULT
  • DUPLICATE_RUN_PROFILES
  • CLUSTER_RUN_PROFILE
runProfileId Run profile to be used from cluster string Required

Remote Engine Gen2

AdvancedRunProfile

Pipeline run profile of type ‘Advanced’
Object
Name Description Type Attributes and examples
name Name string Required
Example
Default run profile
description Description string Optional
Example
Run profile used by default
engineId Pipeline engine id string Optional
Example
5f7eff3245ecf259a1c56daa
definition Json with definition string Optional
Example
Example {"id": 1}
pushdownPreferenceEnabled Prefer executions in SQL Pushdown mode when possible (if pipelines are compatible, they are sent to the data warehouse for execution). boolean Optional
warehouseName Optional warehouse name you can provide to overwrite the default warehouse parameter used to connect to the data warehouse at runtime. string Optional
createDate Creation date of the run profile datetime Optional
FormatRFC3339
updateDate Update date of the run profile datetime Optional
FormatRFC3339
id run profile id string Optional
Example
57f64991e4b0b689a64feed0
type string Optional
Example
advanced

BigDataRunProfile

Pipeline run profile of type ‘Big Data’
Object
Name Description Type Attributes and examples
name Name string Required
Example
Default run profile
description Description string Optional
Example
Run profile used by default
engineId Pipeline engine id string Optional
Example
5f7eff3245ecf259a1c56daa
distribution Distribution string Optional
Example
EMR 5.x (Hadoop 2.7)
executionFramework Execution framework string Optional
Example
Spark on Yarn
microBatchInterval Micro-batch interval (in ms) integer Required
FormatINT32
Example
1000
streamingTimeout Streaming timeout (in ms) integer Required
FormatINT32
Example
10
yarnQueue Yarn queue string Optional
Example
q1
driverCores Driver cores integer Optional
FormatINT32
Example
1
driverMemory Driver cores (in MB) integer Optional
FormatINT32
Example
512
temporaryStorage Temporary storage string Required
Example
/tmp-storage
yarnStrategy YarnStrategy Required
executorMemoryOverhead Execution memory overhead (MB) integer Required
FormatINT32
Example
384
isCheckpointingEnabled Checkpointing boolean Optional
checkPointFilePath Checkpoint file path. Required if checkpointing enabled string Optional
Example
/file-path
parameters Parameters ObjectOptional
id Run profile id string Optional
Example
57f64991e4b0b689a64feed0
type Type string Optional
Enum
  • standard
  • big-data
  • databricks
  • advanced
createDate Created on datetime Optional
FormatRFC3339
updateDate Updated on datetime Optional
FormatRFC3339

Databrick

Object
Name Description Type Attributes and examples
endpoint Endpoint string Required
Example
https://endpoint
token Token (can also be provided in the Remote Engine configuration) string Optional
Example
1
stagingLocation DBFS library staging directory string Required
Example
dbfs:/staging/
target DatabricksCluster Required
cloudProvider Cloud provider string Required
Enum
  • AWS
  • Azure
Example
AWS

DatabricksCluster

Object
Name Description Type Attributes and examples
cluster New or existing cluster string Required
Enum
  • new
  • old
Example
new
id ID of existing cluster. Required for ‘old’ cluster string Optional
Example
132553
sparkVersion Databricks runtime version. Required for ‘new’ cluster string Optional
Example
5.5.x-scala2.11
nodeTypeId Node type ID. Required for ‘new’ cluster string Required
Example
m4.large
logsDestination Logs destination. Required for ‘new’ cluster string Required
Example
dbfs:/cluster-logs
workersNumber Number of workers. Required for ‘new’ cluster integer Required
FormatINT32
Example
2

DatabricksRunProfile

Pipeline run profile of type ‘Databricks’
Object
Name Description Type Attributes and examples
name Name string Required
Example
Default run profile
description Description string Optional
Example
Run profile used by default
engineId Pipeline engine id string Optional
Example
5f7eff3245ecf259a1c56daa
microBatchInterval Micro-batch interval (in ms) integer Required
FormatINT32
Example
1000
databricks Databrick Optional
parameters Parameters ObjectOptional
id Run profile id string Optional
Example
57f64991e4b0b689a64feed0
type Type string Optional
Enum
  • standard
  • big-data
  • databricks
  • advanced
createDate Created on datetime Optional
FormatRFC3339
updateDate Updated on datetime Optional
FormatRFC3339

DataIntegrationRunProfile

Object
Name Description Type Attributes and examples
name string Required
description string Optional
jvmArguments array of string Optional
Datatype details
Type Description Attributes and examples
array jvm arguments applied on job execution
Example
["-XX:PermSize=128m", "-XX:MaxHeapFreeRatio=70"]
string

DataIntegrationRunProfileResponse

Object
Name Description Type Attributes and examples
id string Optional
Example
"62444fbce9f05d51f8da4031"
name string Required
Example
"Default run profile"
description string Optional
Example
"run profile description"
engineId string Optional
Example
"6246ea2275ca6f0e05bc4a87"
runtimeId string Optional
Example
"f4422f10-4152-4a3f-adae-3b97ba3ba918"
jvmArguments string Optional
Example
["-XX:MaxHeapFreeRatio=70", "-XX:PermSize=128m"]
type string Optional
Enum
  • data-integration
createDate date Optional
updateDate string Optional

PipelineEngine

Object
Name Description Type Attributes and examples
id Resource id string Required
Example
5cdd2ce2c737973f9d581b98
name Resource name string Required
Example
Remote Engine
description Resource description string Optional
Example
Some description
workspace WorkspaceInfo Optional
createDate Date of creation of the resource datetime Required
FormatRFC3339
updateDate Date of updating of the resource datetime Optional
FormatRFC3339
runtimeId Resource runtime id string Required
Example
9e8acf72-00d5-451b-9175-97bd44cd6b13
availability Availability status of engine|cluster string Optional
Enum
  • AVAILABLE
  • NOT_AVAILABLE
  • RETIRED
Example
NOT_AVAILABLE
managed Indicates whether target runtime (engine/cluster) is managed or not boolean Optional
status string Optional
Enum
  • PAIRED
  • NOT_PAIRED
preAuthorizedKey string Optional
cloudRunner boolean Optional
runStatus Engine status for Cloud Runner string Optional
Enum
  • STARTING
  • RUNNING
  • STOPPING
  • STOPPED
  • START_FAILED
  • STOP_FAILED
  • ERROR
  • NOT_FOUND

PipelineEngineRequest

Object
Name Description Type Attributes and examples
name Engine name string Required
Example
Remote Engine
environmentId Id of engine environment string Required
Example
5c1111d7a4186a4eafed0587
workspaceId Id of engine workspace string Optional
Example
5c2222d7a4186a4eafed0587
description Engine|cluster description string Optional
Example
Some description

PipelineRunProfileResponse

Pipeline run profile response object.
Object
Name Description Type Attributes and examples
name Name string Required
Example
Default run profile
description Description string Optional
Example
Run profile used by default
engineId Pipeline engine id string Optional
Example
5f7eff3245ecf259a1c56daa
id Run profile id string Optional
Example
57f64991e4b0b689a64feed0
type Type string Optional
Enum
  • standard
  • big-data
  • databricks
  • advanced
  • data-integration
createDate Created on datetime Optional
FormatRFC3339
updateDate Updated on datetime Optional
FormatRFC3339

StandardRunProfile

Pipeline run profile of type ‘Standard’
Object
Name Description Type Attributes and examples
name Name string Required
Example
Default run profile
description Description string Optional
Example
Run profile used by default
engineId Pipeline engine id string Optional
Example
5f7eff3245ecf259a1c56daa
configuration Configuration Required
executionFramework Execution framework string Required
Example
Spark local
pushdownPreferenceEnabled Prefer executions in SQL Pushdown mode when possible (if pipelines are compatible, they are sent to the data warehouse for execution). boolean Optional
warehouseName Optional warehouse name you can provide to overwrite the default warehouse parameter used to connect to the data warehouse at runtime. string Optional

StandardRunProfileResponse

Pipeline run profile of type ‘Standard’
Object
Name Description Type Attributes and examples
name Name string Required
Example
Default run profile
description Description string Optional
Example
Run profile used by default
engineId Pipeline engine id string Optional
Example
5f7eff3245ecf259a1c56daa
configuration Configuration Required
executionFramework Execution framework string Required
Example
Spark local
id Run profile id string Optional
Example
57f64991e4b0b689a64feed0
type Type string Optional
Enum
  • standard
  • big-data
  • databricks
  • advanced
createDate Created on datetime Optional
FormatRFC3339
updateDate Updated on datetime Optional
FormatRFC3339
pushdownPreferenceEnabled Prefer executions in SQL Pushdown mode when possible (if pipelines are compatible, they are sent to the data warehouse for execution). boolean Optional
warehouseName Optional warehouse name you can provide to overwrite the default warehouse parameter used to connect to the data warehouse at runtime. string Optional

YarnStrategy

Object
Name Description Type Attributes and examples
type Yarn strategy type string Required
Enum
  • Fixed
  • Dynamic
initialExecutors Initial number of executors (for Dynamic strategy) integer Optional
FormatINT32
minExecutors Min. number of executors (for Dynamic strategy) integer Optional
FormatINT32
maxExecutors Max. number of executors (for Dynamic strategy) integer Optional
FormatINT32
coresPerExecutor Cores per executor integer Optional
FormatINT32
executorMemory Executor memory (in MB) integer Optional
FormatINT32
executorsNumber Number of executors (for Fixed strategy) integer Optional
FormatINT32

Create new Pipeline Engine

POST /processing/runtimes/pipeline-engines
Create new Pipeline Engine

Request

Body
application/json
PipelineEngineRequest

Response

201Status 201
Created
Body
application/json
PipelineEngine
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Get Pipeline Engines

GET /processing/runtimes/pipeline-engines
Get Pipeline Engines

Request

Query parameters
Name Description Type Attributes and examples
query search query (FIQL format), e.g. “workspace.environment.id==5cb47ca4b1b5247f6006529e”,“status==PAIRED” string Optional

Response

200Status 200
OK
Body
application/json
details
Name Description Type Attributes and examples
array of PipelineEngine
Datatype details
Type Description Attributes and examples
array
PipelineEngine
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Delete Pipeline Engine by id

DELETE /processing/runtimes/pipeline-engines/{engineId}
Delete Pipeline Engine by id

Request

Path variables
Name Description Type Attributes and examples
engineId pipeline engine id string Required

Response

204Status 204
No Content
400Status 400
Bad Request
Body
*/*
ErrorResponse
401Status 401
Unauthorized
Body
*/*
ErrorResponse
403Status 403
Forbidden
Body
*/*
ErrorResponse
404Status 404
Not Found
Body
*/*
ErrorResponse
409Status 409
Conflict
Body
*/*
ErrorResponse
500Status 500
Internal Server Error
Body
*/*
ErrorResponse

Get Pipeline Engine by id

GET /processing/runtimes/pipeline-engines/{engineId}
Get Pipeline Engine by id

Request

Path variables
Name Description Type Attributes and examples
engineId pipeline engine id string Required

Response

200Status 200
OK
Body
application/json
PipelineEngine
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Unpair Pipeline Engine

DELETE /processing/runtimes/pipeline-engines/{engineId}/pairing
Unpair Pipeline Engine

Request

Path variables
Name Description Type Attributes and examples
engineId pipeline engine id string Required

Response

200Status 200
OK
Body
text/plain
string
400Status 400
Bad Request
Body
text/plain
ErrorResponse
401Status 401
Unauthorized
Body
text/plain
ErrorResponse
403Status 403
Forbidden
Body
text/plain
ErrorResponse
404Status 404
Not Found
Body
text/plain
ErrorResponse
500Status 500
Internal Server Error
Body
text/plain
ErrorResponse

Get run profile for engine

GET /processing/runtimes/pipeline-engines/{engineId}/run-profiles

Request

Path variables
Name Description Type Attributes and examples
engineId run profile id string Required

Response

200Status 200
successful operation
Body
application/json
details
Name Description Type Attributes and examples
array of PipelineRunProfileResponse
Datatype details
Type Description Attributes and examples
array
PipelineRunProfileResponse
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal server error
Body
application/json
ErrorResponse

Create new advanced run profile

POST /processing/runtimes/pipeline-engines/{engineId}/run-profiles/advanced

Request

Path variables
Name Description Type Attributes and examples
engineId engine id string Required
Body
application/json
AdvancedRunProfile

Response

201Status 201
Created
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json
ErrorResponse
500Status 500
Internal Server Error
Body
application/json
ErrorResponse

Update advanced run profile

PUT /processing/runtimes/pipeline-engines/{engineId}/run-profiles/advanced/{runProfileId}

Request

Path variables
Name Description Type Attributes and examples
engineId engine id string Required
runProfileId run profile id string Required
Body
application/json
AdvancedRunProfile

Response

204Status 204
No Content
400Status 400
Bad Request
Body
*/*
ErrorResponse
401Status 401
Unauthorized
Body
*/*
ErrorResponse
403Status 403
Forbidden
Body
*/*
ErrorResponse
404Status 404
Not Found
Body
*/*
ErrorResponse
500Status 500
Internal Server Error
Body
*/*
ErrorResponse

Get advanced run profile

GET /processing/runtimes/pipeline-engines/{engineId}/run-profiles/advanced/{runProfileId}

Request

Path variables
Name Description Type Attributes and examples
engineId engine id string Required
runProfileId run profile id string Required

Response

200Status 200
OK
Body
application/json
AdvancedRunProfile
400Status 400
Bad Request
Body
application/json
ErrorResponse
401Status 401
Unauthorized
Body
application/json
ErrorResponse
403Status 403
Forbidden
Body
application/json
ErrorResponse
404Status 404
Not Found
Body
application/json