Shape analysis [VC 21.3 GEN]

Shapes can be analyzed to detect for example detect cropping and silence.

Analysis will also generation information for generating waveform data, to be visualized in a UI.

From version 5.4 and onward, sending an empty analyze request document will not perform an analysis of the shape. In earlier versions an empty document meant performing an analysis for all parameters, which could lead to excessive database disk usage.

Operations

Analyze a specific shape with job parameters in the request body

POST /item/(item-id)/shape/(shape-id)/analyze

Analyzes the specified shape with the parameters specified in the job document. The result of the analyze will appear in the bulky metadata of the shape when doing a transcoder analysis, or in the metadata of the item when doing a cognitive service analysis.

Query Parameters:	resourceId (string) – The transcoder or cognitive service resource to use to execute the analysis. storageId (string) – The storage on which to store a temporary analysis data file when using a Vidinet transcoder to analyze a shape. If no storage id has been specified Vidispine will (by default) automatically pick a supported storage. The storage id will be ignored when using a non Vidinet transcoder. callbackId (string) – The callback resource id to use for finding and running callback scripts. notification (string) – The placeholder job notification to use for this job. notificationData (string) – Any additional data to include for notifications on this job. priority (string) – The priority to assign to the job. Default is `MEDIUM` . jobmetadata (string[]) – Additional information for the job task.
Accepts:	application/xml, application/json – AnalyzeJobDocument
Produces:	application/xml, application/json – JobDocument
Role:	_job_write

Example

Analyze a shape checking only for black frames:

NONE

POST /item/VX-123/shape/VX-456/analyze
Content-Type: application/xml

<AnalyzeJobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
    <black>
        <threshold>0.1</threshold>
    </black>
</AnalyzeJobDocument>

HTML/XML

<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-426</jobId>
  <user>admin</user>
  <started>2012-03-26T11:27:49.173Z</started>
  <status>READY</status>
  <type>ANALYZE</type>
  <priority>MEDIUM</priority>
</JobDocument>

Example

Analyze a shape with custom parameters:

NONE

POST /item/VX-124/shape/VX-457/analyze
Content-Type: application/xml

<AnalyzeJobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <black>
    <threshold>0.1</threshold>
    <percentage>95</percentage>
  </black>
  <freeze>
    <time>1.0</time>
    <threshold>0.05</threshold>
  </freeze>
  <bars>
    <percentage>10</percentage>
    <threshold>0.05</threshold>
  </bars>
</AnalyzeJobDocument>

HTML/XML

<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-427</jobId>
  <user>admin</user>
  <started>2012-03-26T11:27:49.173Z</started>
  <status>READY</status>
  <type>ANALYZE</type>
  <priority>MEDIUM</priority>
</JobDocument>

In this example, settings for black frame, freeze and bar detection are included. The threshold elements determine the threshold to use when detecting black frames or freezes. The values have the following meaning:

threshold for black frame detection and bar detection denotes that any pixel whose value is greater than threshold * 255 should not be regarded as black. I.e. only if threshold is 0 will only completely black pixels be counted.
For freeze frame detection threshold determines how much any one pixel may change between two frames. If the difference in value between two frames is greater than threshold * 255, the frame will not be regarded as frozen.

Example

Analyze a shape with audio parameters:

NONE

POST /item/VX-124/shape/VX-457/analyze
Content-Type: application/xml

<AnalyzeJobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
    <channel stream="1"/>
    <channel stream="2"/>
</AnalyzeJobDocument>

HTML/XML

<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-428</jobId>
  <user>admin</user>
  <started>2012-03-26T11:47:12.565Z</started>
  <status>READY</status>
  <type>ANALYZE</type>
  <priority>MEDIUM</priority>
</JobDocument>

In this example, only the audio of the shape is analyzed. And specifically streams number 1 and 2. The numbers correspond to the essenceStreamId of the audio component. To analyze all streams, just add a single channel and omit the stream attribute.

Analyze a specific shape using an analyze preset

POST /item/(item-id)/shape/(shape-id)/analyze

Analyzes the specified shape with the parameters from the analyze preset specified. The result of the analyze will appear in the bulky metadata of the shape when doing a transcoder analysis, or in the metadata of the item when doing a cognitive service analysis.

Query Parameters:	preset (string) – The analyze preset to use for the job resourceId (string) – The transcoder or cognitive service resource to use to execute the analysis. storageId (string) – The storage on which to store a temporary analysis data file when using a Vidinet transcoder to analyze a shape. If no storage id has been specified Vidispine will (by default) automatically pick a supported storage. The storage id will be ignored when using a non Vidinet transcoder. callbackId (string) – The callback resource id to use for finding and running callback scripts. notification (string) – The placeholder job notification to use for this job. notificationData (string) – Any additional data to include for notifications on this job. priority (string) – The priority to assign to the job. Default is `MEDIUM` . jobmetadata (string[]) – Additional information for the job task.
Produces:	application/xml, application/json – JobDocument
Role:	_job_write

Analyze an item

POST /item/(item-id)/analyze

Analyzes an item with the parameters specified in the job document. The result of the analyze will appear in the bulky metadata of the shape when doing a transcoder analysis, or in the metadata of the item when doing a cognitive service analysis.

New in version 5.0.

Query Parameters:	resourceId (string) – The transcoder or cognitive service resource to use to execute the analysis. tag (string) – The shape tag to analyze. If omitted the original shape tag will be used. storageId (string) – The storage on which to store a temporary analysis data file when using a Vidinet transcoder to analyze a shape. If no storage id has been specified Vidispine will (by default) automatically pick a supported storage. The storage id will be ignored when using a non Vidinet transcoder. callbackId (string) – The callback resource id to use for finding and running callback scripts. notification (string) – The placeholder job notification to use for this job. notificationData (string) – Any additional data to include for notifications on this job. priority (string) – The priority to assign to the job. Default is `MEDIUM` . jobmetadata (string[]) – Additional information for the job task.
Accepts:	application/xml, application/json – AnalyzeJobDocument
Produces:	application/xml, application/json – JobDocument
Role:	_job_write

Viewing the results

The results of the analysis is stored in the bulky metadata for the shape. For example, to view the black frame information (if available), go to /item/VX-123/shape/VX-456/metadata/bulky/black. You should see something like the following:

HTML/XML

<BulkyMetadataDocument xmlns="http://xml.vidispine.com/schema/vidispine" id="VX-295">
  <field stream="0" end="6@50" start="0@50">
    <key>black</key>
    <value>1</value>
  </field>
  <field stream="0" end="1650@50" start="1490@50">
    <key>black</key>
    <value>1</value>
  </field>
</BulkyMetadataDocument>

Each field contains a start and an end attribute, denoting the start and end timecodes for the black frames.

Loudness analysis

When an analysis is done, a loudness analysis is done automatically. The result of the loudness analysis is written to the bulky metadata, but there are utility methods to easily extract the information.

Get loudness values

GET /item/(item-id)/loudness

Extracts loudness information from bulky metadata.

Produces:	application/xml, application/json – LoudnessDocument
Role:	_item_shape_read

Example

NONE

POST /item/VX-124/shape/VX-457/analyze

NONE

GET /item/VX-124/loudness

HTML/XML

<LoudnessDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-124</id>
  <shape>VX-457</shape>
  <shapeTag>original</shapeTag>
  <mix>
    <name>Left</name>
    <weightdB>0.0</weightdB>
    <sourceStream>0</sourceStream>
    <sourceChannel>0</sourceChannel>
  </mix>
  <mix>
    <name>Right</name>
    <weightdB>0.0</weightdB>
    <sourceStream>0</sourceStream>
    <sourceChannel>1</sourceChannel>
  </mix>
  <mix>
    <name>Center</name>
    <weightdB>0.0</weightdB>
  </mix>
  <mix>
    <name>Left Surround</name>
    <weightdB>1.5</weightdB>
  </mix>
  <mix>
    <name>Right Surround</name>
    <weightdB>1.5</weightdB>
  </mix>
  <startLoudness>0@48000</startLoudness>
  <endLoudness>1339200@48000</endLoudness>
  <startRange>0@48000</startRange>
  <endRange>1296000@48000</endRange>
  <loudnessLU>0.014140396527686505</loudnessLU>
  <loudnessRangeLU>4.974758665644899</loudnessRangeLU>
</LoudnessDocument>

Get loudness values for interval

PUT /item/(item-id)/loudness

Extracts loudness information from bulky metadata. Start and end range can be specified, as well as custom mixing.

Accepts:	application/xml, application/json – LoudnessDocument
Produces:	application/xml, application/json – LoudnessDocument
Role:	_item_shape_read

Waveform information

The waveform data is not exactly the waveform, but measurements of the RMS values with a rather high sampling rate.

Get waveform data

GET /item/(item-id)/waveform/values

Returns the waveform data as a WaveformDataDocument.

Query Parameters:	itemTrack (string) – The `itemTrack` value of the audio channel within the shape. stream (string) – The stream value of the audio channel within the component of the shape. channel (string) – The channel value of the audio channel within the stream of the component. If `itemTrack` and `stream` are omitted, this value can be used to denote tracks in a linear fashion, regardless of `itemTrack` and `stream`. Then `channel=0` means the first audio track, channel=1` the second, etc. Default is `0`. shape (string) – The shape id to use to get information from. If omitted the shape tag will be used. Note that an analysis of this shape must be done before the information is available. tag (string) – The shape tag to use. Default is `original`. start (string) – The start time code to get waveform information for. Default is `-INF`. end (string) – The end time code to get waveform information for. Default is `+INF`. dB (boolean) – `true` - Return RMS dB values. `false` (default) - Return RMS 1-based absolute values. width (integer) – The number of sample points to return. Default is `400`.
Produces:	application/xml, application/json – WaveformDataDocument
Role:	_item_shape_read

Get waveform data (deprecated)

Deprecated since version 4.5.1.

GET /item/(item-id)/waveform/data

Returns the waveform data as a JSON array.

Query Parameters:	itemTrack (string) – The `itemTrack` value of the audio channel within the shape. stream (string) – The stream value of the audio channel within the component of the shape. channel (string) – The channel value of the audio channel within the stream of the component. If `itemTrack` and `stream` are omitted, this value can be used to denote tracks in a linear fashion, regardless of `itemTrack` and `stream`. Then `channel=0` means the first audio track, channel=1` the second, etc. Default is `0`. shape (string) – The shape id to use to get information from. If omitted the shape tag will be used. Note that an analysis of this shape must be done before the information is available. tag (string) – The shape tag to use. Default is `original`. start (string) – The start time code to get waveform information for. Default is `-INF`. end (string) – The end time code to get waveform information for. Default is `+INF`. dB (boolean) – `true` - Return RMS dB values. `false` (default) - Return RMS 1-based absolute values. width (integer) – The number of sample points to return. Default is `400`.
Produces:	application/json – A JSON array with one JSON object. The JSON object contains one value with key `data`. The value is a JSON array with `width` number of data points.
Role:	_item_shape_read

Get waveform image

GET /item/(item-id)/waveform/image

Returns an image with the waveform drawn on the canvas as described by the query parameters.

Query Parameters:	itemTrack (string) – The `itemTrack` value of the audio channel within the shape. stream (string) – The stream value of the audio channel within the component of the shape. channel (string) – The channel value of the audio channel within the stream of the component. If `itemTrack` and `stream` are omitted, this value can be used to denote tracks in a linear fashion, regardless of `itemTrack` and `stream`. Then `channel=0` means the first audio track, channel=1` the second, etc. Default is `0`. shape (string) – The shape id to use to get information from. If omitted the shape tag will be used. Note that an analysis of this shape must be done before the information is available. tag (string) – The shape tag to use. Default is `original`. start (string) – The start time code to get waveform information for. Default is `-INF`. end (string) – The end time code to get waveform information for. Default is `+INF`. dB (boolean) – `true` - Return RMS dB values. `false` (default) - Return RMS 1-based absolute values. width (integer) – The number of sample points to return. Default is `400`. height (integer) – The height, in pixels, of the image. Default is 100. bgcolor (string) – The background color of the image, as hex triplet. Default is `#000000` (black). fgcolor (string) – The color of the waveform, as hex triplet. Default is `#ffffff` (white). hgridline (string) – The position of primary horizontal gridlines, in units of the audio. Default is `""` (no gridline). hgridlinecolor (string) – The color of primary horizontal gridlines. Default is `#808080`. hgridline2 (string) – The position of secondary horizontal gridlines, in units of the audio. Default is `""` (no gridline). hgridline2color (string) – The color of secondary horizontal gridlines. Default is `#404040` vgridline (string) – The position of primary vertical gridlines, where 0 is left border and 1 is right border. Default is `""` (no gridline). vgridlinecolor (string) – The color of primary vertical gridlines. Default is `#808080` vgridline2 (string) – The position of primary vertical gridlines, where 0 is left border and 1 is right border. Default is `""` (no gridline). vgridline2color (string) – The color of primary vertical gridlines. Default is `#404040` min (number) – The audio value that corresponds the bottom border. Defaults to `-1` if `dB` is `false`, and `-80` otherwise. max (number) – The audio value that corresponds the top border. Defaults to `1` if `dB` is `false`, and `0` otherwise.
Produces:	image/png – A PNG image.
Role:	_item_shape_read

Get waveform image URI

GET /item/(item-id)/waveform/imageURI

Returns a URI that does not require authentication to the generated image. The URI expires after 1 hour.

Query Parameters:	itemTrack (string) – The `itemTrack` value of the audio channel within the shape. stream (string) – The stream value of the audio channel within the component of the shape. channel (string) – The channel value of the audio channel within the stream of the component. If `itemTrack` and `stream` are omitted, this value can be used to denote tracks in a linear fashion, regardless of `itemTrack` and `stream`. Then `channel=0` means the first audio track, channel=1` the second, etc. Default is `0`. shape (string) – The shape id to use to get information from. If omitted the shape tag will be used. Note that an analysis of this shape must be done before the information is available. tag (string) – The shape tag to use. Default is `original`. start (string) – The start time code to get waveform information for. Default is `-INF`. end (string) – The end time code to get waveform information for. Default is `+INF`. dB (boolean) – `true` - Return RMS dB values. `false` (default) - Return RMS 1-based absolute values. width (integer) – The number of sample points to return. Default is `400`. height (integer) – The height, in pixels, of the image. Default is 100. bgcolor (string) – The background color of the image, as hex triplet. Default is `#000000` (black). fgcolor (string) – The color of the waveform, as hex triplet. Default is `#ffffff` (white). hgridline (string) – The position of primary horizontal gridlines, in units of the audio. Default is `""` (no gridline). hgridlinecolor (string) – The color of primary horizontal gridlines. Default is `#808080`. hgridline2 (string) – The position of secondary horizontal gridlines, in units of the audio. Default is `""` (no gridline). hgridline2color (string) – The color of secondary horizontal gridlines. Default is `#404040` vgridline (string) – The position of primary vertical gridlines, where 0 is left border and 1 is right border. Default is `""` (no gridline). vgridlinecolor (string) – The color of primary vertical gridlines. Default is `#808080` vgridline2 (string) – The position of primary vertical gridlines, where 0 is left border and 1 is right border. Default is `""` (no gridline). vgridline2color (string) – The color of primary vertical gridlines. Default is `#404040` min (number) – The audio value that corresponds the bottom border. Defaults to `-1` if `dB` is `false`, and `-80` otherwise. max (number) – The audio value that corresponds the top border. Defaults to `1` if `dB` is `false`, and `0` otherwise.
Produces:	text/plain – The URI to the image.
Role:	_item_shape_read

Get waveform images for all audio channels

GET /item/(item-id)/waveform/alltracks

Solely used for debugging. May be deprecated in newer releases.

Returns a HTML document including image references to waveform images for all channels. Query parameters can be used to control the image appearance.

Query Parameters:	itemTrack (string) – The `itemTrack` value of the audio channel within the shape. stream (string) – The stream value of the audio channel within the component of the shape. channel (string) – The channel value of the audio channel within the stream of the component. If `itemTrack` and `stream` are omitted, this value can be used to denote tracks in a linear fashion, regardless of `itemTrack` and `stream`. Then `channel=0` means the first audio track, channel=1` the second, etc. Default is `0`. shape (string) – The shape id to use to get information from. If omitted the shape tag will be used. Note that an analysis of this shape must be done before the information is available. tag (string) – The shape tag to use. Default is `original`. start (string) – The start time code to get waveform information for. Default is `-INF`. end (string) – The end time code to get waveform information for. Default is `+INF`. dB (boolean) – `true` - Return RMS dB values. `false` (default) - Return RMS 1-based absolute values. width (integer) – The number of sample points to return. Default is `400`. height (integer) – The height, in pixels, of the image. Default is 100. bgcolor (string) – The background color of the image, as hex triplet. Default is `#000000` (black). fgcolor (string) – The color of the waveform, as hex triplet. Default is `#ffffff` (white). hgridline (string) – The position of primary horizontal gridlines, in units of the audio. Default is `""` (no gridline). hgridlinecolor (string) – The color of primary horizontal gridlines. Default is `#808080`. hgridline2 (string) – The position of secondary horizontal gridlines, in units of the audio. Default is `""` (no gridline). hgridline2color (string) – The color of secondary horizontal gridlines. Default is `#404040` vgridline (string) – The position of primary vertical gridlines, where 0 is left border and 1 is right border. Default is `""` (no gridline). vgridlinecolor (string) – The color of primary vertical gridlines. Default is `#808080` vgridline2 (string) – The position of primary vertical gridlines, where 0 is left border and 1 is right border. Default is `""` (no gridline). vgridline2color (string) – The color of primary vertical gridlines. Default is `#404040` min (number) – The audio value that corresponds the bottom border. Defaults to `-1` if `dB` is `false`, and `-80` otherwise. max (number) – The audio value that corresponds the top border. Defaults to `1` if `dB` is `false`, and `0` otherwise.
Produces:	text/html – A HTML document.
Role:	_item_shape_read

Highlights extraction

New in version 5.4.

Utilizing Nablet Shrynk, an analysis job can calculate an “interest factor” for each frame in a video. This data can then be used to create a highlight reel. There are a number of AI models built in, and in order to get good results it is important to use a model that is tailored to your content. Currently, these models are available:

football
ice-hockey
basketball
formula1
handball

Example

Analyze a shape with default parameters:

NONE

POST /item/VX-123/shape/VX-456/analyze
Content-Type: application/xml

<AnalyzeJobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <highlighter>
    <model>football</model>
  </highlighter>
</AnalyzeJobDocument>

HTML/XML

<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-429</jobId>
  <user>admin</user>
  <started>20120-08-20T11:27:49.173Z</started>
  <status>READY</status>
  <type>ANALYZE</type>
  <priority>MEDIUM</priority>
</JobDocument>

Get a highlight reel EDL

GET /item/(item-id)/shape/(shape-id)/highlighter-edl

If a highligheter analysis has been performed, this returns a conform document for producing a highlight reel of the desired duration.

Query Parameters:	duration (integer) – The desired duration of the highlight reel in seconds.
Produces:	application/xml, application/json – ConformDocument
Role:	_item_read

Start a highlight reel creation job

POST /item/(item-id)/shape/(shape-id)/highlight-render

Starts a highlight render job for the given shape, producing an output file of the desired duration.

Query Parameters:	duration (integer) – The desired duration of the highlight reel in seconds. conformMetadata (boolean) – `true` (default) - Copy metadata from the source items, according to the timeline, to the resulting item. `false` - Do not copy metadata from the source items. sourceTag (string) – Comma-separated list of shape tags, that specify the shapes that should be used as inputs. original (string) – If specified, should be one of the tags specified in the tag parameter. Specifies that the original shape tag will be reset to the shape created to this tag. resourceId (string) – The transcoder resource to use to execute the transcode. tag (string) – Comma-separated list of shape tags, that specify the desired output. createThumbnails (boolean) – `true` (default) - Creates thumbnails according to default transcoder rules. `false` - No thumbnails will be created. createPosters (string) – A list of time codes to use for creating posters. thumbnailService (string) – Identifies which thumbnail resource that should be used. destinationItem (string) – An item id, to which the new new shape will be associated. notification (string) – The placeholder job notification to use for this job. notificationData (string) – Any additional data to include for notifications on this job. priority (string) – The priority to assign to the job. Default is `MEDIUM` . jobmetadata (string[]) – Additional information for the job task.
Produces:	application/xml, application/json – JobDocument
Role:	_job_write

Smart cropping

New in version 5.4.

Utilizing Nablet Heightscreen, landscape format content can be cropped into portrait mode, using an AI algorithm that automatically determines the areas of highest interest in the video.

Example

Analyze a shape with default parameters:

NONE

POST /item/VX-123/shape/VX-456/analyze
Content-Type: application/xml

<AnalyzeJobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <smartcrop>
    <aspect>9:16</aspect>
  </smartcrop>
</AnalyzeJobDocument>

HTML/XML

<JobDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <jobId>VX-430</jobId>
  <user>admin</user>
  <started>20120-08-20T12:12:42.758Z</started>
  <status>READY</status>
  <type>ANALYZE</type>
  <priority>MEDIUM</priority>
</JobDocument>

Get a smart cropping EDL

GET /item/(item-id)/shape/(shape-id)/smartcrop-edl

If a smartcrop analysis has been performed, this returns a sequence document for producing an output with the desired aspect ratio.

Query Parameters:	aspect (string) – The aspect ratio to use, substituting “_” for “:” (e.g. `9_16`). An analyze job for this aspect ratio must have been run prior to making this request.
Produces:	application/xml, application/json – SequenceDocument
Role:	_item_read

Start a smart cropping job

POST /item/(item-id)/shape/(shape-id)/smartcrop-render

Starts a smart cropping render job, producing an output with the desired aspect ratio.

Query Parameters:	aspect (string) – The aspect ratio to use, substituting “_” for “:” (e.g. `9_16`). An analyze job for this aspect ratio must have been run prior to making this request. tag (string[]) – The shape tag specifying the format of the rendered sequence. sourceTag (string[]) – The shape tag specifying the shapes to use as input. original (string) – If specified, should be one of the tags specified in the tag parameter. Specifies that the original shape tag will be reset to the shape created to this tag. destinationItem (string) – An item id, to which the new new shape will be associated. resourceId (string) – The transcoder resource to use to execute the transcode. New in version 4.16. notification (string) – The placeholder job notification to use for this job. notificationData (string) – Any additional data to include for notifications on this job. priority (string) – The priority to assign to the job. Default is `MEDIUM` . jobmetadata (string[]) – Additional information for the job task.
Produces:	application/xml, application/json – JobDocument
Role:	_job_write