S3 Event SNS Notifications [VC 21.3.1 GEN]

New in version 5.0.

Scanning an S3 storage can be expensive both in terms of time and money. To make it cheaper to access an S3 bucket, you can configure Vidispine to receive S3 events from an Amazon SNS topic and increase the time between regular storage scans, which are more expensive.

Prerequisites

Assuming that you already have an S3 storage setup in Vidispine, the next step is to create an SNS topic and configure the S3 bucket to send events to that topic. This configuration is made entirely within AWS and instructions on how to configure it can be found here: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

• The two types of events that Vidispine are interested in are:

  • ObjectCreated:* (All object create)

  • ObjectRemoved:* (All object delete)

  New in version 21.3.

  • ObjectsRestore:Completed (Object restored from archive)

• The Vidispine endpoint that handles SNS messages requires authentication, so the url that is registered as a subscriber in AWS must have the credentials encoded in it. Because of this the Vidispine API must be accessible via HTTPS. Unencrypted HTTP is not supported.

• Use one SNS topic per bucket. Don’t send events from multiple buckets to the same topic, as this is not supported by Vidispine.

Close restored files faster

New in version 21.3.

Faster update of file state (CLOSED) when a file is restored from archive.

Configuration

1. Set the storage method metadata snsTopic to the ARN of the SNS topic. Vidispine will only accept SNS messages from this topic.

   NONE

   PUT /storage/VX-1/method/VX-2/metadata/snsTopic
   Content-Type: text/plain
   
   arn:aws:sns:eu-west-1:123456791011:topic_name

   NONE

   GET /storage/VX-1

   HTML/XML

   <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
   <StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
     ...
     <method>
       <uri>s3://bucketname/</uri>
       ...
       <metadata>
         <field>
           <key>snsTopic</key>
           <value>arn:aws:sns:eu-west-1:123456791011:topic_name</value>
         </field>
       </metadata>
     </method>
     ...
   </StorageDocument>

2. Configure Vidispine as a subscriber for this topic in AWS. The endpoint that receives SNS message is /API/sns-endpoint, and as this endpoint requires authentication the credentials must be encoded in the url. A complete url for subscription might look like this: https://<username>:<password>@<address-to-vidispine>/API/sns-endpoint or https://sns-user:my-password@example.myvidispine.com/API/sns-endpoint.

• The role _file_write is required to call the SNS endpoint. You can use a pre-existing user or create a new user specifically for the SNS subscription, as long as it has this role.

• Vidispine will automatically confirm the SNS subscription request that is sent by AWS when the subscription is created, as long as the topic matches the storage method metadata snsTopic. Checking that the subscription has been confirmed is a good way of verifying that everything has been configured correctly.

1. Finally you can configure Vidispine to do regular scans of the storage less often by setting the storage property scanInterval. Vidispine will perform a storage scan every scanInterval second, so setting this to 3600 will make Vidispine scan it once every hour. See When are files scanned? for more information.