Transcoders [VC 21.3 GEN]
When you import items the Vidispine transcoder will be used to detect the type of media that is being imported and, of course, to transcode the media to any formats that you have requested.
The common operations performed by the transcoder are:
Media shape deduction
Partial file extraction
XMP extraction and rewrite
The Vidispine transcoder has a REST API that Vidispine uses to perform the above operations. This API is not described in this document, as it typically should not be accessed directly.
Adding a transcoder
Add a transcoder by creating a new
transcoder resource. The resource document should contain information on how to reach the transcoder and what storages the transcoder has direct access to.
POST /resource/ Content-Type: application/xml <?xml version="1.0"?> <ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine"> <transcoder> <url>http://transcoder.example.com:8888/</url> <directAccess> <filter>file:/srv/media/.*</filter> </directAccess> </transcoder> </ResourceDocument>
Vidispine checks the status of transcoders continuously in the background. As such, if the configuration is correct you will see that the transcoder shows up as online in a few seconds.
<?xml version="1.0"?> <ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine"> <id>VX-7</id> <transcoder> <url>http://transcoder.example.com:8888/</url> <directAccess> <filter>file:/srv/media/.*</filter> </directAccess> <state>ONLINE</state> </transcoder> </ResourceDocument>
The Vidispine installer will by default install and configure a transcoder in Vidispine for you, so this step is typically not needed.
Using multiple transcoders
Depending on your license, you may be allowed to use more than one transcoder. To do so, simply add additional transcoders as explained above. Vidispine will submit transcode jobs to the transcoder based on the current number of jobs being processed by the transcoder.
Vidispine will use the transcoder with the least amount of work. If a transcoder goes offline then any transcode job steps using that transcoder will fail and be retried using one of the online transcoders. If all transcoders are offline then jobs will wait for one to become available.
clusterName property must be set if multiple Vidispine installations are to share a transcoder. Each installation must have a unique cluster name. This applies regardless if the installations have the same site name or not.
How transcoders perform jobs
A transcoder will perform a job as soon as it is received, and will not schedule jobs for later execution. Vidispine, that is, the user of the transcoder is responsible for scheduling which jobs a transcoder should execute and when they should be executed.
Transcoder job limit
maxJob setting can be used to limit the number of Vidispine jobs that may use a specific transcoder at the same time. If all transcoders are busy, jobs will be put on
WAITING state, with a
TranscoderBusy problem. The jobs will restart as soon as any qualified transcoder becomes available again.
<ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine"> <id>VX-7</id> <transcoder> <url>http://transcoder.example.com:8888/</url> <maxJob>5<maxJob> </transcoder> </ResourceDocument>
A Vidispine job typically triggers multiple transcoder jobs, e.g. shape deductions and transcodes, so in the above case there may still be more than 5 running jobs on a transcoder. They will however all belong to at most 5 jobs in Vidispine.
The above setting works for VSA transcoders as well. VSA users can also add
transcoder.maxJob to one of the agent configuration files. For example:
$ cat /etc/vidispine/agent.conf.d/transcoder.conf transcoder.maxJob=5
The transcoder’s configuration file
The transcoder configuration file
config.xml contains default settings for the transcoder and need typically not be modified, as the settings can instead by configured in Vidispine.
Modify the transcoder file
On a Linux system, copy the file
/etc/transcoder-config.xml. Then edit
/etc/transcoder-config.xml. The file in
/etc takes precedence over the file in
Modify the transcoder resource
On all operating systems, the transcoder configuration can be changed by adding configuration to the resource definition of the transcoder (Adding a transcoder).
Note that port of the transcoder cannot be changed in this fashion.
Modifying the transcoder configuration is this fashion takes precedence over the local configuration file and the global transcoder configuration, see below.
Modify all transcoders
It is also possible to change the configuration of all transcoders, by setting the configuration property
transcoderDefaultConfiguration to the XML representation of the transcoder configuration.
The preferred way of changing the thumbnail and poster settings is by changing the appropriate values in the TranscodePresetDocument in a shape tag. For example, by changing the
thumbnailPeriod elements. The setting in shape tag have priority over the transcoder setting.
thumbnailResolution element contains the default resolution of the thumbnails produced by the transcoder.
<a:thumbnailResolution> <a:width>320</a:width> <a:height>240</a:height> </a:thumbnailResolution>
You can also change the thumbnailing frequency by changing
thumbnailPeriod. For example, to thumbnail every 3 seconds:
<a:thumbnailPeriod> <a:samples>3</a:samples> <a:timeBase> <a:numerator>1</a:numerator> <a:denominator>1</a:denominator> </a:timeBase> </a:thumbnailPeriod>
If the transcoder does not use a scene change detection plugin, the frequency defaults to once every 10 seconds.
To have the transcoder send metrics to a StatsD server you can either:
Enable StatsD using the API, see StatsD
Update the transcoder configuration with the address and port of the StatsD server:
<a:statsd> <a:destination> <a:address>127.0.0.1</a:address> <a:port>8125</a:port> </a:destination> <a:prefix>t1</a:prefix> </a:statsd>
prefix element configures the prefix to use for each metric. By default this is the
Transcoder resources settings
Path to temporary storage
Controls where temporary files are stored. Default is
/tmp on UNIX-like systems, or
%TEMP% on Windows.
Number of decoding threads
Controls the number of decoding threads. Defaults to
4 for I-frame-only formats. The actual number of threads used depends on codec.
Number of encoding threads
Controls the number of encoding threads. Defaults to automatic setting. The actual number of threads used depends on codec.
HTTP buffer sizes
Controls the size of HTTP reads and writes of the transcoder.
dataBufferSize controls the maximum number of read bytes in memory. Default is 100 MB.
dataBufferWriteSize controls the maximum number of write bytes in memory. Default is 100 MB.
dataBufferFlushTime controls the number of seconds written bytes are stored in memory before it is flushed. Default is 4 seconds.
For system that uses segment files and where the transcoder has enough memory, it is recommended to increase these numbers, up to 10 times.
To control the memory and disk usage used by the transcoder for image processing, use the
<imagemagick> element in the transcoder configuration. The most important settings are listed below, for a complete list, see http://www.imagemagick.org/script/resources.php (under environment variables, used without the
MAGICK_ prefix in the transcoder configuration).
Maximum heap usage
<a:imagemagick> <a:key>MEMORY_LIMIT</a:key> <a:value>1GB</a:value> </a:imagemagick>
Temporary work area
<a:imagemagick> <a:key>TEMPORARY_PATH</a:key> <a:value>/var/tmp</a:value> </a:imagemagick>
The default value is the value set by the general transcoder temporary path, see above. It is recommended that the
tempPath setting is used, rather than the
The following is done to remove the need to configure the transcoders directly:
Vidispine pushes its own license to the transcoder, so that each transcoder does not need a license file of their own.
The transcoder returns the IP address from where the license was pushed, that is, the IP address of the application server, removing the need for explicitly configuring the reverse address, that is, where the transcoder can reach Vidispine, in most cases.
In addition, Vidispine generates temporary pre-authorized URIs that are used by the transcoder. This removes the need for entering any application server information in the transcoder configuration file.
Reverse address and NAT
The reverse address does not work if there is NAT or other port forwarding mechanisms between the application server and the transcoder. If so, the address to VS-EA can be overridden in the definition for the transcoder by setting the
<?xml version="1.0"?> <ResourceDocument xmlns="http://xml.vidispine.com/schema/vidispine"> <transcoder> <url>http://transcoder.example.com:8888/</url> <reverseAddress>vs.example.com</reverseAddress> </transcoder> </ResourceDocument>
The rules for how the address to Vidispine is determined are as follows:
If the configuration property
apiNoauthUriis set, it is used for all transcoders.
If the configuration property
apiNoauthPortis set, it is used for together with the detected or manually set reverse address.
Transcoder’s access to media
By default, the transcoder accesses non-file-schema media through the application server. This has several advantages:
The same user is used for all file access.
Possibility for support for extended file attributes and permissions.
Support for other file systems (URI schemes).
Streaming the media puts some extra load on the application server. Some tuning might be necessary.
The transcoder resource in Vidispine can be set up to access files directly. By adding a
directAccess element to the transcoder resource, Vidispine will let the transcoder access the media directly. If no
directAccess elements are present, an implicit
<directAccess> <filter>file:.*</filter> </directAccess>
is added. In order to tell Vidispine that all files should go via the application server, add an
<directAccess> <filter>NO_MATCH</filter> <!-- dummy regular expression that does not match anything --> </directAccess>
For all file systems that supports read-while-write, and for container formats that are built for streaming (e.g. MXF), growing file is supported when streamed through the application server. If growing files is required to local files with the
file scheme, a
NO_MATCH element as per above must be added to the resource configuration.