S3 Event SQS Notifications [VC 21.3.1 GEN]

Scanning an S3 storage can be expensive both in terms of time and money. To make it cheaper to access an S3 bucket, you can configure Vidispine to poll an Amazon SQS queue for S3 events and increase the time between regular storage scans, which are more expensive.

Prerequisites

Assuming that you already have an S3 storage setup in Vidispine, the next step is to create an SQS queue and configure the S3 bucket to send events to that queue. This configuration is made entirely within AWS and instruction on how to configure it can be found here: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

The three types of events that Vidispine are interested in are:
- ObjectCreated:* (All object create)
- ObjectRemoved:* (All object delete)
New in version 21.3.
- ObjectsRestore:Completed (Object restored from archive)
Vidispine will connect to the SQS queue using the credentials from the S3 storage method URI, so that user must have access to both the bucket and the queue. For the SQS queue the user needs permission for the following actions on the queue:
- sqs:GetQueueUrl
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:DeleteMessageBatch
- sqs:PurgeQueue
Use one SQS queue per bucket. Don’t send events from multiple buckets to the same queue, as this is not supported by Vidispine.

Use IAM roles

New in version 21.3.

It is now possible to use IAM roles to handle SQS queue notifications across accounts.

This feature will use stsRegion and use that region when making the call to the STS API for the assume role request.
RoleSessionName is an optional field, if unset Vidicore will generate one automatically.
Please note that the role that Vidicode will try to assume must have the same SQS permissions as above.

Close restored files faster

New in version 21.3.

Faster update of file state (CLOSED) when a file is restored from archive.

To make Vidicore close files faster when they are restored from archive, make sure that ObjectsRestore:Completed is checked in Event types of the storage’s notifications.

Configure the storage

To have Vidispine poll a SQS queue instead of scanning a S3 bucket, set the storage method metadata sqsName and sqsEndpoint to enable this feature:

NONE

PUT /storage/VX-1/method/VX-2/metadata/sqsName
Content-Type: text/plain

s3-event-queue

NONE

PUT /storage/VX-1/method/VX-2/metadata/sqsEndpoint
Content-Type: text/plain

sqs.eu-west-1.amazonaws.com

New in version 21.3.

NONE

PUT /storage/VX-1/method/VX-2/metadata/roleArn
Content-Type: text/plain

arn:aws:iam::<accountId>:role/<roleName>

NONE

(optional)
PUT /storage/VX-1/method/VX-2/metadata/roleExternalId
Content-Type: text/plain

external-id

NONE

(optional)
PUT /storage/VX-1/method/VX-2/metadata/roleSessionName
Content-Type: text/plain

role-session-name

NONE

GET /storage/VX-1

HTML/XML

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<StorageDocument xmlns="http://xml.vidispine.com/schema/vidispine">
  ...
  <method>
    <uri>s3://bucketname/</uri>
    ...
    <metadata>
      <field>
        <key>sqsName</key>
        <value>s3-event-queue</value>
      </field>
      <field>
        <key>sqsEndpoint</key>
        <value>sqs.eu-west-1.amazonaws.com</value>
      </field>
      <field>
        <key>roleArn</key>
        <value>arn:aws:iam::<accountId>:role/<roleName></value>
      </field>
      <field>
        <key>roleExternalId</key>
        <value>external-id</value>
      </field>
      <field>
        <key>roleSessionName</key>
        <value>role-session-name</value>
      </field>
    </metadata>
  </method>
  ...
</StorageDocument>

Then make sure that the storage metadata scanOnStart is true (this is the default).

Due to the distributed nature of Amazon SQS, the messages come unordered. On every start up, Vidispine will need to purge the queue and do a full scan of the storage, to sync the file list with database.
Finally you can configure Vidispine to do regular scans of the storage less often by setting the storage property scanInterval. Vidispine will perform a storage scan every scanInterval second, so setting this to 3600 will make Vidispine scan it once every hour. See When are files scanned? for more information.

You can check the storage method status (lastSuccess, lastFailure, failureMessage) to determine if the configuration is correct or not. For example, if a non-existing queue is specified:

HTML/XML

<failureMessage>
   Error polling SQS: The specified queue does not exist for this wsdl version. (...)
</failureMessage>