How to setup auto-import rules on S3 storage

Amazon S3 (Simple Storage Service) is an object storage with a web service interface to store and retrieve any amount of data. Here we will show you how to configure an auto-import rule on your S3 storage.

There are 3 different main steps in this guide;

• Automatically discover files on S3

• Create items for them

• Create proxies on /srv/media

1. Create an AWS S3 storage (or another storage supporting the S3 method).

   Your storage must support autoimport and the showImportables property in the StorageDocument must be set to true.  (<showImportables>true</showImportables)

   The files on the storage will then be discovered and available for use with GET /storage/importable or GET /storage/auto-import/.
   http://apidoc.vidispine.com/latest/storage/storage.html#storage-schemes
   

2. Put the SQS metadata fields on that S3 storage (not on /srv/media storage).

3. Create a storage rule that puts all shapes with tag __mp4 on the /srv/media storage;
   

   CODE

   POST /API/shape-tag/__mp4/storage-rule
   
   <StorageRuleDocument xmlns="http://xml.vidispine.com/schema/vidispine">
     <storage>VX-1</storage>  <!-- the /srv/media storage -->
   </StorageRuleDocument>

   http://apidoc.vidispine.com/latest/storage/storage-rule.html

4. Create an auto-import rule and assign it to the S3 storage;

   CODE

   <AutoImportRuleDocument xmlns="http://xml.vidispine.com/schema/vidispine">
           <tag>__mp4</tag>
   </AutoImportRuleDocument>

   http://apidoc.vidispine.com/latest/storage/auto-import.html

5. If you also want the original file to be copied to /srv/media, add another storage rule, this time to the original shape-tag.
   Repeat step 3 above, but this time with the following call;

   CODE

   POST /API/shape-tag/original/storage-rule

   
   

6. If you want the original file to be removed from the S3 storage, you have two options;
   a) Add a negative storage rule to the original shape-tag storage rule. That will say, original files will always be stored on the /srv/media storage, but never on the S3 storage.
   b) Set lowWatermark=highWatermark=0 on the S3 storage. That will make Vidispine always trying to move files away from that storage. Vidispine will not move files that do not belong to an item, or files that have not been put somewhere else.
   http://apidoc.vidispine.com/latest/storage/storage-rule.html#negative-rule-example