Storage reference [VC 21.3 GEN]

Manage storages.

GET /storage Retrieves the storages that have been configured. Query Parameters: size (string) – Range of bytes, in format s1-s2. Returns storages with nominal size that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages. freebytes (string) – Range of bytes, in format s1-s2. Returns storages with free space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages. usedbytes (string) – Range of bytes, in format s1-s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Size units can be used. Default is -, all storages. freeamount (string) – Range of percent as integers, in format s1-s2. Returns storages with used space that is in that range. Either number can be omitted to not specify lower/upper limit. Default is -, all storages. files (string) – Range of files as integers, in format s1-s2. Returns storages with number of files that is in that range. Either number can be omitted to not specify lower/upper limit. Default is -, all storages. storagegroup (string[]) – List of storage groups. storage-group - Returned storage is member of specified storage group. -storage-group - Returned storage is not member of specified storage group. Default is to return all storages. status (string[]) – List of storage status. status - Returned storage has this status. -status - Returned storage does not have this status. Default is to return all storages. url (string) – Returns storages with a method matching this URL. May include wildcards * and ?. Produces: application/xml, application/json – StorageListDocument text/plain – CRLF-delimited list of storage ids Role: _storage_read

Example

GET /storage

<StorageListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <storage>
    <id>VX-1</id>
    <state>NONE</state>
    <type>LOCAL</type>
    <capacity>1600000000000</capacity>
    <freeCapacity>19993896424</freeCapacity>
    <method>
      <id>VX-6</id>
      <uri>file:///mnt/main/</uri>
      <bandwidth>0</bandwidth>
      <read>true</read>
      <write>true</write>
      <browse>true</browse>
      <lastSuccess>2014-07-04T08:39:00.739+02:00</lastSuccess>
      <type>NONE</type>
    </method>
    <metadata/>
    <lowWatermark>0</lowWatermark>
    <highWatermark>0</highWatermark>
    <showImportables>true</showImportables>
  </storage>
  <storage>
    <id>VX-2</id>
    <state>NONE</state>
    <type>LOCAL</type>
    <capacity>15000000000</capacity>
    <freeCapacity>19548305962</freeCapacity>
    <method>
      <id>VX-4</id>
      <uri>file:///mnt/ingest/</uri>
      <read>true</read>
      <write>true</write>
      <browse>true</browse>
      <lastSuccess>2014-07-04T08:38:56.773+02:00</lastSuccess>
      <type>NONE</type>
    </method>
    <metadata/>
    <lowWatermark>0</lowWatermark>
    <highWatermark>200000000000</highWatermark>
    <showImportables>true</showImportables>
  </storage>
</StorageListDocument>

Create a storage

POST /storage Creates a new storage. Accepts: application/xml, application/json – StorageDocument Produces: application/xml, application/json – StorageDocument text/plain – Storage id Role: _storage_write

If your storage is to be used for autoimport, the showImportables property in the StorageDocument must be set to true. The files on the storage will then be discovered and available for use with GET /storage/importable or GET /storage/auto-import/.

Example

POST /storage
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <method>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
  </method>
  <lowWatermarkPercentage>90</lowWatermarkPercentage>
  <highWatermarkPercentage>75</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>112500000000</lowWatermark>
  <highWatermark>135000000000</highWatermark>
  <lowWatermarkPercentage>75</lowWatermarkPercentage>
  <highWatermarkPercentage>90</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Retrieve a storage

GET /storage/(storage-id) Retrieves a specific storage. Produces: application/xml, application/json – StorageDocument Role: _storage_read

Example

GET /storage/VX-2

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>112500000000</lowWatermark>
  <highWatermark>135000000000</highWatermark>
  <lowWatermarkPercentage>75</lowWatermarkPercentage>
  <highWatermarkPercentage>90</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Update a storage

PUT /storage/(storage-id) Updates an existing storage. Accepts: application/xml, application/json – StorageDocument Produces: application/xml, application/json – StorageDocument text/plain – Storage id

Example

PUT /storage/VX-2
Content-Type: application/xml

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <lowWatermarkPercentage>80</lowWatermarkPercentage>
  <highWatermarkPercentage>95</highWatermarkPercentage>
</StorageDocument>

<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <id>VX-2</id>
  <state>NONE</state>
  <type>LOCAL</type>
  <capacity>150000000000</capacity>
  <freeCapacity>150000000000</freeCapacity>
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <type>NONE</type>
  </method>
  <metadata/>
  <lowWatermark>120000000000</lowWatermark>
  <highWatermark>142500000000</highWatermark>
  <lowWatermarkPercentage>80</lowWatermarkPercentage>
  <highWatermarkPercentage>95</highWatermarkPercentage>
  <showImportables>true</showImportables>
</StorageDocument>

Delete a storage

DELETE /storage/(storage-id) Deletes the storage. All files in storage will remain after call, but the Vidispine system will no longer manage them. Query Parameters: safe (boolean) – true - Storage will only be deleted if there are no files connected to items on the storage. false (default) - Storage will be deleted, and any file entities will be disconnected from items and deleted. Status Codes: 409 Conflict – If safe parameter is true this error code will be given if there are files connected to items on the storage. Role: _storage_write

Storage information

Update the storage type

PUT /storage/(storage-id)/type/(type) Sets the type of the storage. Role: _storage_write

Retrieve the storage status

GET /storage/(storage-id)/status Retrieves the status of the storage. Produces: text/plain – Status string Role: _storage_read

Retrieve the amount of free space on a storage

GET /storage/(storage-id)/freespace Retrieves the amount of free space on the storage. Produces: text/plain – Amount of free space as decimal number, between 0 and 100, inclusive Role: _storage_read

Rescanning

Rescan a storage

POST /storage/(storage-id)/rescan Triggers a rescan of a single storage. The scanInterval property can be used to control how often (in seconds) a storage is scanned. Default is 60. By setting scanInterval to -1 storage scans will be disabled. By calling /rescan, the system is forced to rescan a storage without delay. Query Parameters: path (string) – Rescans a specific path on storage to find importable files. This will not update deleted files. New in version 5.4.

Storage methods

Storage methods specify the different ways of accessing files on a specific storage.

Credentials are encrypted. This means that passwords cannot be viewed through the API/server logs.

List storage methods

GET /storage/(storage-id)/method Retrieves the access methods configured on a specific storage. Query Parameters: url (string) – Only return methods with this URL. Wildcards (?, *) can be used, for example http:*. read (boolean) – true - Only return methods which have file read capability. false (default) - Return methods regardless of file read capability. write (boolean) – true - Only return methods which have file write capability. false (default) - Return methods regardless of file write capability. browse (boolean) – true - Only return methods which have file browse capability. false (default) - Return methods regardless of file browse capability. Produces: application/xml, application/json – StorageMethodListDocument Role: _storage_read

Example

GET /storage/VX-2/method

<StorageMethodListDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  <method>
    <id>VX-4</id>
    <uri>file:///mnt/ingest/</uri>
    <bandwidth>0</bandwidth>
    <read>true</read>
    <write>true</write>
    <browse>true</browse>
    <lastSuccess>2014-07-04T09:55:09.779+02:00</lastSuccess>
    <type>NONE</type>
  </method>
</StorageMethodListDocument>

Update or create a storage method

PUT /storage/(storage-id)/method Adds a new access method to the storage. If URL matches an existing method, a new method is not created, instead the existing one is updated. Query Parameters: url (string) – Required. Method is accessed through this URL read (boolean) – true (default) - Method has file read capability false - Method does not have file read capability write (boolean) – true (default) - Method has file write capability false - Method does not have file write capability browse (boolean) – true (default) - Method has file browse capability false - Method does not have file browse capability bandwidth (integer) – The bandwidth of this method in bytes per second. Default 0. type (string) – method type. Default is empty. Produces: text/plain – Tabbed tuples : id, URL, status string of method Role: _storage_write

Example

PUT /storage/VX-2/method?url=http://10.12.0.3/ingest/&read=true&write=false&browse=false

VX-5        http://10.12.0.3/ingest/        NONE

Retrieve a storage method

GET /storage/(storage-id)/method/(method-id) Retrieves a specific access method to storage. Produces: application/xml, application/json – StorageMethodListDocument Role: _storage_read

Create/update a storage method

PUT /storage/(storage-id)/method/(method-id) Updates access method to the storage. Query Parameters: url (string) – Required. Method is accessed through this URL read (boolean) – true - Method has file read capability. false - Method does not have file read capability. write (boolean) – true - Method has file write capability. false - Method does not have file write capability. browse (boolean) – true - Method has file browse capability. false - Method does not have file browse capability. bandwidth (integer) – The bandwidth of this method in bytes per second. Default 0. type (string) – method type. Produces: text/plain – Tabbed tuples : id, URL, status string of method Role: _storage_write

Delete a storage method

DELETE /storage/(storage-id)/method/(method-id) DELETE /storage/(storage-id)/method Removes an access method from a storage. Query Parameters: url (string) – Method is accessed through this URL Role: _storage_write

Importing/exporting a storage definition

Vidispine can export a definition document describing all the files within a storage, which can later be imported to a new storage on a different Vidispine instance.

Export a storage definition

GET /storage/(storage-id)/export Creates a StorageImportDocument that describes every file on the storage. This should be saved to a file which can later be used to import the storage definition. Produces: application/xml, application/json – StorageImportDocument Role: _storage_read

Import a storage definition

POST /storage/import Creates a new storage based on the StorageImportDocument. A file entity will be created for each entry in the document, if a file with that ID does not already exist. Finally, a storage method will be added, with the path supplied in the call. Query Parameters: path (string) – The file system path to where the files are located. Accepts: application/xml, application/json – A StorageImportDocument Role: _storage_write

Evacuating storages

If you would like to delete a storage, but you still have files there which are connected to items, you can first trigger an evacuation of the storage. This will cause Vidispine to attempt to delete redundant files, or move files to other storages. Once the evacuation is complete, the storage will get the state EVACUATED.

Evacuate a storage

PUT /storage/(storage-id)/evacuate Trigger evacuation of a storage. Role: _storage_write

Cancel evacuation of a storage

DELETE /storage/(storage-id)/evacuate Cancel the evacuation process on a storage. Role: _storage_write

Key-value metadata

Storages support key-value metadata.

Reserved keys

Certain keys are used to control the behavior of a storage, they must not be used to store generic metadata.

closeLimit An integer specifying the number of minutes until the system automatically considers an open file to be closed. Default: 5

lostLimit An integer specifying the number of minutes until the system automatically considers a missing file to be lost. Default: 10

toAppearLimit An integer specifying the number of minutes the system waits for a file to appear before considering it to be missing. Default: 10

refreshInterval The interval (in seconds) that Vidispine will compare the list of file seen on disk with the list of files in the database. This to fix any possible mismatch. Default: 900

hashMode Set to false to disable hash computation for this storage. Default: true

additionalHash A comma-separated list of additional hashing algorithms that should be applied to files. E.g.: MD5,SHA-256. The hash algorithm FILENAME is also supported, which takes the last part of the path concatenated with the file size, and calculates a SHA-1 checksum of that string. Default: (none)

fileListBatchSize Maximum size of list of files that are updated in the database at the same time. Default: 100000

keepMissingFiles If set to false then missing files that do not belong to any items will be removed from the database instead of being marked as lost. Default: To the value of the configuration property keepMissingFiles.

keepEmptyDirectories Do not delete empty parent directories when deleting the last file in a directory, see Parent directory management. Default: To the value of the configuration property keepEmptyDirectories.

statsPerSecond An integer specifying the maximum number of stat system calls that are made to the storage (only file:// URIs). See also the configuration property statsPerSecond. Default: (none)

scanOnStart By default, Vidispine will scan the file system on start-up. Storages that should never be scanned, that should solely use SQS or SNS for example, should have this parameter set to false. Default: true Since: 5.0

refreshOnStart By default, Vidispine synchronizes the database with the file system on start-up. On very large systems, this can take a very long time. Set this parameter to false, and use the rescan functionality (see below) instead. Default: true

deleteFileIfNotFound If set to true: if file that is marked for deletion cannot be deleted, but is not found on the file system, it is assumed to be deleted. Default: false

deleteFileIfReadOnly If a file that is marked for deletion cannot be deleted, and the file exists on a read-only storage, then: If set to true: delete the file from the database, as if the file had been deleted. If set to a file state: set the file to this state. Default: false Example: true, ARCHIVED, LOST

excludeFilter A regular expression. Any file with a path (relative to the storage’s root folder) that matches the expression will be ignored. Note that the expression must match the entire path, not only a part of the path. Default: (none)

detectRenamedFiles If this metadata is set to true, whenever a file belonging to an item is marked as LOST, and there is another file on the same storage with the same hash value, the new file is associated with the item instead. Other possible values are all, where the old (lost) file resided on any storage, or a comma-separated list of identifiers of storages on which the old (lost) file resided on. Default: (none)

probeFileBeforeClosing If this metadata is set to true, Vidispine will ensure that OPEN files are readable and are not being modified by any other process before marking them as CLOSED. Jobs with transfer steps will be set to WAITING with a FileUnavailable job problem until the file becomes available. This is useful for Windows storages where the size and metadata is constant while a file is copied to the storage, but the file cannot be read. Default: false

vxaId If this metadata field is set, the storage will be automatically deleted if the share is removed from the Vidispine Server Agent. Default: (none) for regular storages, automatically set by VSA when a share is added and a storage is created.

sqsName The name of Amazon SQS queue to poll for S3 file events. See S3 Event SQS Notifications for more information. This key is only read for S3 methods.

sqsEndpoint The endpoint of Amazon SQS queue. This key is only read for S3 methods. Example: sqs.eu-west-1.amazonaws.com

snsTopic The ARN of an SNS topic to receive S3 file events from. See S3 Event SNS Notifications for more information. Example: arn:aws:sns:eu-west-1:123456791011:topic_name

s3EventETag When receiving file events from SNS this can be specified to set the ETag included in the message from the storage as metadata on the file. Example: true Default: false

s3EventTime When receiving file events from SNS this can be specified to set the timestamp in the message received as metadata on the file. Example: true Default: false

hashingThreadCount The number of hashing thread that should be run on the VSA side. Note: this configuration only work for VSA storage. Default: 1

noDefaultHash Do not calculate hash checksums for files on the storage. Example: true Default: false

verifyHashAfterTransfer If a job is copying or moving a file to this storage the hash of the source file and the hash of the newly copied/moved file are compared to ensure that the new file is binary identical to the original file. If the hashes differ the job fails and the destination file is removed. If set to true: hash verification is performed after the file has arrived on this storage. If set to false: no hash verification is performed. The hash verification procedure makes use of the hash calculation that is happening on all storages by default (see File hashing). If hash calculation is disabled on the source storage the hash is calculated before the copy/move operation is performed. If hash calculation is disabled on the destination storage the hash is calculated after the copy/move operation has completed. These hashes are stored in the Vidispine database for later reuse. Note that hash calculation requires reading the whole file and thus is a lengthy process that can significantly increase execution time of copy/move jobs. Default: false

storageActivationFile Require a .storage file to be present in the storage method’s URI for storage to register as online. Takes precedence over the global setting. Default: (none) - The global setting applies. Since: 4.17

scanMethodAlgorithm Specify which scan algorithm to use, see Storage scanning algorithm. Default: To the value of the configuration property scanMethodAlgorithm. Since: 5.5.2

Size units

In the query parameters above, size can be specified in number of bytes as integer [ [ whitespace ] unit ] where the multiplier unit can be one of (case insensitive):

Unit	Factor
`KB`	1000¹
`K`, `KiB`	2¹⁰
`MB`	1000²
`M`, `MiB`	2²⁰
`GB`	1000³
`G`, `GiB`	2³⁰
`TB`	1000⁴
`T`, `TiB`	2⁴⁰
`PB`	1000⁵
`P`, `PiB`	2⁵⁰
`EB`	1000⁶
`E`, `EiB`	2⁶⁰
`ZB`	1000⁷
`Z`, `ZiB`	2⁷⁰
`YB`	1000⁸
`Y`, `YiB`	2⁸⁰

It should be noted that currently, Vidispine has a 64-bit limit on the size of a storage, which means that multipliers larger than ZB are of limited use.

Managing storages

List all storages

Example

Create a storage

Example

Retrieve a storage

Example

Update a storage

Example

Delete a storage

Storage information

Update the storage type

Retrieve the storage status

Retrieve the amount of free space on a storage

Rescanning

Rescan a storage

Storage methods

List storage methods

Example

Update or create a storage method

Example

Retrieve a storage method

Create/update a storage method

Delete a storage method

Importing/exporting a storage definition

Export a storage definition

Import a storage definition

Evacuating storages

Evacuate a storage

Cancel evacuation of a storage

Key-value metadata

Reserved keys

Size units