Jobs [VC 21.3 GEN]

Jobs make up the long running tasks in Vidispine. They are created in response to requests that would otherwise not be able to respond in time, such as import, export and transcode requests.

The actions performed by a job is determined by its type. Bound to the type are a number of steps, or tasks, defined by the task definitions. The tasks form a graph, and typically execute in sequence, but it is also possible for tasks to start in parallel. This happens for example when importing and transcoding a growing file. The transfer step will initiate the transfer and then trigger the transcode step to start once enough data (the header) from the file has been transferred.

The states of a job are illustrated above. See below for a full description of the states and of the job step states.

Creating jobs

Create jobs by making requests to other RESTful resources:

Concurrency

The number of jobs that execute in parallel is determined by the concurrentJobs configuration property.

Job pools

Using job pools, it is possible to decide how many jobs of different priorities that can run concurrently. Job pools are configured using the job pool configuration resource.

If no pools have been defined then <concurrentJobs> controls the number of concurrent jobs. This is the same setting as the concurrentJobs configuration property. So by default the job pool configuration will look like:

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <concurrentJobs>3</concurrentJobs>
</JobPoolListDocument>

Jobs with priority IMMEDIATE are always started, even if the max concurrent jobs limit is reached. This could impact system performance. To execute the job with IMMEDIATE priority the user must be a super user, that is, have role _super_access_user.

The max concurrent job setting will only have an effect if it is lower then the size of all pools combined.

Priority pools

To start using priority job pools, the job pool definitions must be configured. Priority job pools make it possible to limit the number of concurrent low priority jobs, to make sure that higher priority jobs are able to start even if there are a large number of low priority jobs waiting to be started.

PUT /configuration/job-pool
Content-Type: application/xml

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <pool>
    <priorityThreshold>HIGH</priorityThreshold>
    <size>2</size>
  </pool>
  <pool>
    <priorityThreshold>LOWEST</priorityThreshold>
    <size>3</size>
  </pool>
</JobPoolListDocument>

This configuration will allow at most 3 jobs with a priority of LOWEST to MEDIUM to execute at the same time. It will also allow up to 5 concurrent HIGH/HIGHEST priority jobs, as the second pool will contain jobs with a priority of LOWEST or higher (the priority threshold is the lower bound and pools have no upper priority bound.)

If there is no job pool with a priority threshold that matches low priority jobs then such jobs will not be started. For example, to only let jobs with a priority of MEDIUM or higher to execute:

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <pool>
    <priorityThreshold>MEDIUM</priorityThreshold>
    <size>3</size>
  </pool>
</JobPoolListDocument>

Dedicated pools

New in version 5.4.

To start using a dedicated job pool, the configuration property dedicatedJobPool must be set to true. See create/modify configuration properties for more information. The job pools must also be configured. If no pools have been defined, the default behaviour will occur where the <concurrentJobs> controls the number of concurrent jobs, regardless if the dedicatedJobPool is set to true or not.

Dedicated job pools make it possible to dedicate certain pools to jobs of certain priority. This can be used to make sure that jobs with all priorities are able to start, regardless of how large the number of other priority jobs that are ready to start.

A job pool, with a priority threshold and a size, defines the upper limit of how many jobs of that priority that are allowed to run in that pool. If there are no ready jobs of that priority, those slots will be used for higher priority jobs.

PUT /configuration/job-pool
Content-Type: application/xml

<JobPoolListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <pool>
    <priorityThreshold>HIGH</priorityThreshold>
    <size>2</size>
  </pool>
  <pool>
    <priorityThreshold>MEDIUM</priorityThreshold>
    <size>3</size>
  </pool>
  <pool>
    <priorityThreshold>LOWEST</priorityThreshold>
    <size>5</size>
  </pool>
</JobPoolListDocument>

This configuration dedicates certain pools for specific priority jobs. The size of priority threshold HIGH allows 2 jobs of priority HIGH to run in that pool. If there are no jobs with priority HIGH ready to be started, jobs with priority HIGHEST will use those 2 slots.

The size of priority threshold MEDIUM allows for 3 jobs of priority MEDIUM to run in that pool, concurrently with 2 jobs with priority HIGH, as defined by the first pool. If there are no jobs with priority MEDIUM ready to be started, jobs with priority HIGHEST will use those 3 slots. If there are no jobs with priority HIGHEST ready to be started, then the 3 MEDIUM slots will be used by jobs with priority HIGH.

The last defined size of priority threshold LOWEST allows for 5 jobs with priority LOWEST to run concurrently of HIGH and MEDIUM jobs. If there are no jobs with priority LOWEST ready to be started, then jobs with priority HIGHEST or HIGH or MEDIUM or LOW will start, depending if there are jobs of those priorities ready to start, in that specific order.

If there is no defined job pool with a priority threshold that matches lower priority jobs, then such jobs will not be started. For example, to only let jobs with a priority of MEDIUM or higher to execute, remove the job pool with priority threshold LOWEST from the example above. Then jobs of priority LOW and LOWEST will never run.

Jobs of priority IMMEDIATE are always started and therefore do not need to be defined by a pool.

Job problems

Jobs will enter the state WAITING if a recoverable problem has occurred. Depending on the problem the system might resolve itself or require manual assistance, for example if the system is out of storage space.

A system with no job problems will report:

GET /job/problem HTTP/1.1
Content-Type: application/xml

<JobProblemListDocument xmlns="http://xml.vidispine.com/schema/vidispine"/>

A system where the transcoder is unreachable for some reason may report:

GET /job/problem HTTP/1.1
Content-Type: application/xml

<JobProblemListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <problem>
    <id>31532534</id>
    <type>TranscoderOffline</type>
    <job>VX-172716</job>
  </problem>
</JobProblemListDocument>

There can be multiple jobs waiting for a problem to be resolved, for example, in case of transcoder or storage problems. For JavaScript problems there will however be one problem per job, as the problem condition is defined by a step specific for each job.

Job tasks

The action performed by a task can be implemented either as a method in a Java class or as a JavaScript. Using JavaScript is recommended for all new applications.

POST /task-definition/ HTTP/1.1
Content-Type: application/xml

<TaskDefinitionListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <task>
    <description>A custom JavaScript step</description>
    <script><![CDATA[
// This script does nothing but fail the job
job.fatalFail("Testing job failing");
]]></script>
    <step>10000</step>
    <dependency>
      <previous>false</previous>
      <allPrevious>true</allPrevious>
    </dependency>
    <jobType>PLACEHOLDER_IMPORT</jobType>
    <critical>false</critical>
  </task>
</TaskDefinitionListDocument>

Defining new tasks

See JavaScript tasks on how to create JavaScript tasks.

Task dependencies

The execution order is defined by the step numbers and dependencies of the steps. The dependency element defines which steps a specific step depend on. There is also the parallelDependency element that defines the dependencies that apply if the step is executing as a parallel step.

Visualizing tasks

In order to easily see the dependencies between steps for a particular job type, there is functionality to render the job definition as a graph. In order to render the graph, the Graphviz package is required.

Custom job types

It is possible to define custom job types. Each custom job type is defined with a name and an integer identifier. Both of these must be unique. Task definitions can then be added to the job type, in the same way as described above.

Example

First we create the job type:

POST /task-definition/jobtype/MYCOMPANY_CUSTOM_JOB_TYPE?id=25000 HTTP/1.1

<TaskDefinitionListDocument xmlns="http://xml.vidispine.com/schema/vidispine"/>

Then we can add task definitions to our new job type:

POST /task-definition/ HTTP/1.1
Content-Type: application/xml

<TaskDefinitionListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <task>
    <description>A custom JavaScript step</description>
    <script><![CDATA[
    logger.log("My custom job is running!");
]]></script>
    <step>100</step>
    <dependency>
      <previous>false</previous>
      <allPrevious>true</allPrevious>
    </dependency>
    <jobType>MY_CUSTOM_JOB_TYPE</jobType>
    <critical>false</critical>
  </task>
</TaskDefinitionListDocument>

After this has been done, we can now run the job:

POST /job?type=MY_CUSTOM_JOB_TYPE

<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-29</jobId>
  <user>admin</user><started>2016-05-17T12:38:22.999Z</started>
  <type>MY_CUSTOM_JOB_TYPE</type>
  <status>READY</status>
  <priority>MEDIUM</priority>
</JobDocument>