Using Vantage with Cloud Storage (S3, Google, Azure, ECS, Cloudian)

Application Note

Vantage

Synopsis ........................................................................................................................2

Configuring Cloud Access ............................................................................................4

Cloud-Accessible Workflow Actions............................................................................9

Creating a Workflow...................................................................................................12

Note: This guide is written for video professionals who are familiar with using Vantage. To

implement applications in Vantage, you should know how to create workflows and submit jobs. If

you aren’t familiar with Vantage, we suggest that you review the Vantage User’s Guide and Vantage

Domain Management Guide as needed.

Using Cloud Storage with

Vantage

This App Note

applies to

Vantage

Version 7.1 - 8.0

& later

Using Cloud Storage with Vantage

Synopsis

Vantage

Synopsis

Vantage interfaces very smoothly with a number of Cloud Storage systems that provide

cloud-based object storage that can be accessed from Vantage workflows.

Systems covered by this note include the major cloud storage systems:

• Amazon S3

• Google Cloud

• Microsoft Azure

Also covered are software-defined object storage systems that serve as front ends to

virtual storage on cloud storage systems:

• DELL EMC ECS

• Cloudian HyperStore

Note: This note describes the use of cloud storage with on premises Vantage

installations, but it does not describe Vantage Cloud Port. For more information about

using Vantage Cloud Port to run Vantage actions in the cloud, see the Vantage Cloud

Port User Guide at Telestream.net. Vantage Cloud Port adds many advantages when

working with cloud object storage.

Using Cloud Storage with Vantage

Synopsis

Vantage

Vantage with Cloud Storage Overview

Vantage on premises workflows can watch media files in cloud storage, localize

detected new files, perform processing, and then deploy files back to cloud storage.

Important Note: Cloud /Object storage cannot be used as a Vantage Managed Store.

Vantage Managed stores may only be windows-accessible storage locations.

• In the watch phase, you can use Vantage’s Watch and Associate actions to detect

new files in cloud storage.

• In the localize/ingest phase, the workflow copies the media and related files from

the cloud to the local system.

• After localization, processing can occur—typically, transcoding to new media for-

mats and related tasks.

• Workflows can also deliver the output media and related files to a target folder in

the cloud using Copy, Move, Deploy and Delete actions.

Using Cloud Storage with Vantage

Configuring Cloud Access

Vantage

Telestream Software and License Requirements

You can integrate cloud storage into workflows for file ingest and delivery in any

supported Vantage domain without any special Telestream licensing requirements.

Note: Telestream does not recommend a specific cloud security policy and

Telestream does not recommend how you configure permissions in your object store.

You are responsible for granting these permissions to Vantage services in accordance

with your organization’s cloud security policy.

Configuring Cloud Access

Each cloud storage system has unique configuration requirements for working with

Vantage. The following links to documentation will guide you in starting and managing

your cloud storage accounts. Most importantly, you must enable read-only permissions

to allow Vantage to monitor and download objects in storage or read-write permissions

to allow Vantage to read from and write objects to storage.

Amazon S3 Storage Documentation

The following link points to Amazon S3 documentation. Follow the guidance in the

documentation to establish your storage account and enable third party software, such

as Vantage, to access it:

https://docs.aws.amazon.com/s3/?id=docs_gateway

See also the Telestream document, IAM Policies for Vantage AWS S3 Workflows, at

Telestream.net, which explains how to set S3 IAM permissions.

Google Cloud Storage Documentation

The following link points to Google Cloud Storage documentation. Follow the guidance

in the documentation to establish your storage account and enable third party

software, such as Vantage, to access it:

https://cloud.google.com/storage/docs/

For Vantage to read, browse, download, and write (upload) files to and from storage,

the Storage Object Admin role is required.

To read, browse, and download, (but not upload), the Storage Object Viewer role is

sufficent.

The Storage Object Creator role is not recommended because it allows Vantage only to

upload but not to browse or download.

Using Cloud Storage with Vantage

Configuring Cloud Access

Vantage

Microsoft Azure Storage Documentation

The following link points to Microsoft Azure Storage documentation. Follow the

guidance in the documentation to establish your storage account and enable third

party software, such as Vantage, to access it:

https://docs.microsoft.com/en-us/azure/storage/

Telestream recommends using Shared Access Signatures (SAS) to grant limited access

to objects in your storage account to Vantage, without exposing your account key. See

this link for details: https://docs.microsoft.com/en-us/azure/storage/common/storage-

dotnet-shared-access-signature-part-1.

The following figure shows the Azure SAS token settings required for reading from and

writing to Azure Blob Storage from Vantage. For a read-only configuration used only to

monitor a folder, uncheck Write, Delete, and Create.

Using Cloud Storage with Vantage

Configuring Cloud Access

Vantage

Dell EMC ECS Documentation

The following link points to Dell EMC ECS Storage documentation. Follow the guidance

in the documentation to establish your storage account and enable third party

software, such as Vantage, to access it:

https://www.dellemc.com/en-us/storage/ecs/index.htm#accordion0&tab0=1&tab1=3

Cloudian Hyperstore Documentation

The following link points to Cloudian Hyperstore Storage documentation. Follow the

guidance in the documentation to establish your storage account and enable third

party software, such as Vantage, to access it:

https://cloudian.com/resource/

Using Cloud Storage with Vantage

Verifying Access

Vantage

Verifying Access

Before using cloud storage with Vantage, Telestream recommends that you verify your

access to the target buckets and folders independently of Vantage. Install a cloud

server browser such as Cyberduck (free at https://cyberduck.io/download/) on the

Vantage domain server where your workflow services run.

Note: If you have a distributed Vantage domain (a Vantage array), test access to your

ECS system directly from the server where the Vantage service executing your actions

are hosted.

To verify cloud access, use the cloud storage browser (Cyberduck shown) to log in.

1. Launch the cloud browser, open a connection, and select the cloud server type

(Amazon S3 shown).

2. Enter your URL, access key ID and secret key (or other credentials as required) and

click Connect.

Using Cloud Storage with Vantage

Verifying Access

Vantage

If the connection is successful, you can view the objects in the storage file system:

If the connection fails, troubleshoot the problem and retry until the connection

succeeds.

Using Cloud Storage with Vantage

Cloud-Accessible Workflow Actions

Vantage

Cloud-Accessible Workflow Actions

Vantage provides several actions you can use in your Vantage workflows to ingest

(read) media and other files from the cloud, and other actions to deliver (write) files to

the cloud, as well as delete them.

For details on any action in a workflow, open its Inspector in Workflow Designer and

display the man page—click the M icon in the upper right corner of the panel.

Using Watch | Associate Actions to Ingest Media Files

Both actions are part of the Monitor group, and are executed by the Monitor Service.

A Watch action continually monitors the target directory on the specified file system for

new files - you can choose to monitor for media files or other files associated with a

media file - referred to as an attachment. A Watch action initiates a job each time a new

media file is discovered and passes control to the next action in the workflow.

Associate actions are designed to make additional files accessible to the workflow after

the job has been started by an origin action. Associate actions are not origin actions—

they cannot be the first action in a workflow, because they are not designed to submit

jobs. After the job has started (with a Watch action, or a manual or SDK submission, for

example), the Associate action blocks the job from continuing execution until it has

identified the metadata or other type of file (as specified by file selection options you

set). It ingests the file and assigns the nickname you specify so the file can be utilized by

downstream actions, and then passes control to the next action in the workflow.

You configure Watch and Associate actions to access media on Azure, Google, AWS and

many other files systems using their specific connectors in the Select File System dialog

of the action. These actions can also access media in ECS or Cloudian by using the

Generic S3 connector.

The Monitor service that executes a given workflow’s Watch and/or Associate actions

must have read access to the cloud storage bucket or blob to function properly.

Refer to the man page for the applicable file system (S3, Azure, Google) within Vantage

for more specifics regarding the permissions and attributes required for each particular

filesystem/storage type.

Copy Action to Localize Files

Object storage systems are not designed to operate like the Windows file system that

Vantage is designed to work with. Therefore, files on your cloud system must be

localized using a Vantage Copy action. Once the file is stored locally on the Windows

server, Vantage can process the file.

Using Cloud Storage with Vantage

Cloud-Accessible Workflow Actions

Vantage

Using Copy | Move | Deploy | Delete Actions to Deliver Media

Vantage also provides several actions to perform file utility tasks: duplicating or moving

files (deleting the source after a successful write) to a target folder on the destination

cloud storage system, and, in the case of Deploy, delivering one or more files to the

destination.

To configure a Copy and Move action to deliver media to the cloud, you must first

create a Vantage Folder in the Vantage address book and configure it to identify your

target folder. Then, you can select this folder when configuring the action.

The benefit of using a Copy or Move action to deliver a file is that you can dynamically

recreate the folder path when delivering the file, and you can also dynamically create

the file name. See the man page for details.

You use the Deploy action to write files to the cloud by selecting Export and the specific

file system connector.

Note: For ECS, Cloudian or other similar object storage systems that interface to S3,

select the Generic S3 connector.

The Delete action is configured with the nickname of the target file to perform its

task—the nickname always maintains the current location of the file it represents.

Colocate Action

When Vantage Post Producer CML documents contain file references to S3 locations,

Vantage Colocate resolves the external references, localizes the media and updates the

CML document with the local file location. Vantage transcoding actions reference

media by using Windows or UNC pathnames. When content is stored in locations that

require alternative protocols, such as FTP, HTTP, or S3, it is necessary to migrate the

content to a windows accessible location.

Colocate is configured by specifying these settings:

• The output location: This is the local, block store file system (VDOMAIN D: drive).

Specific folders or Vantage stores are valid choices. However, selecting an Available

Vantage Store makes the workflow portable between systems.

• The source file storage location: In this case choose S3.

• S3 credentials for accessing the store: These are the AWS Access key and Secret

Access Keys that were provided when your account was created.

Using Cloud Storage with Vantage

Cloud-Accessible Workflow Actions

Vantage

Colocate Handling of CML Documents

The CML document includes references to the source content that is stored in S3

bucket locations. S3 is object storage, and therefore there are no randomly accessible

blocks to address. Before transcoding can take place, the file must be moved in its

entirety from S3 to a block store location. Vantage Virtual Domains provide block store

(the D: drive) that is directly attached to the VDOMAIN server.

Execution of the Colocate Action results in localization of the media in the Vantage

Store, located on local block storage, and a CML file with updated references:

<?xml version="1.0" encoding="utf-8"?>

<Composition name="" version="1.0" created="2014-08-14T12:48:22.5055655-04:00"

xmlns="Telestream.Soa.Facility.Playlist">

</Source>

</Source>

</Source>

</Source>

<?xml version="1.0" encoding="utf-8"?>

<Composition name="" version="1.0" created="2014-08-14T12:48:22.5055655-04:00"

xmlns="Telestream.Soa.Facility.Playlist">

</Source>

</Source>

</Source>

</Source>

Using Cloud Storage with Vantage

Creating a Workflow

Vantage

Creating a Workflow

This topic illustrates a simple example of how you can create and configure a Vantage

workflow to ingest media from a cloud folder, localize and transcode the file, and then

deliver the new file back to another folder in cloud storage.

Files ingested by this workflow are submitted to Vantage by the Watch action. When a

new file is detected, it ingests the file and starts a job to execute the workflow,

localizing and transcoding the ingested file into a new video using the Flip64 action.

Then, the new file is delivered back to the cloud storage in another folder:

Configuring the Watch Action

When you activate the workflow, the Watch action utilizes the cloud interface (such as

S3) to poll the specified storage for new files. For each new file detected, it ingests the

file and starts a Vantage job to execute the workflow.

The Generic S3 component enables interaction with S3 endpoints which are not

specifically associated with Amazon AWS S3 storage, providing general-purpose

interaction with storage providers that utilize S3 for access.

Note: You can also access your cloud storage using the Associate action, depending

on your workflow requirements. For example, you might need to pick up a sidecar file

and pass it through the workflow with your media file.

To configure the Watch action, open the Watch inspector and follow these steps:

1. Click Watch for > Media Files, and select Any Media.

2. In the Select File System panel, select a file system, such as S3 AWS, Google Cloud,

Microsoft Azure, or Generic S3 (for ECS or Cloudian).

Using Cloud Storage with Vantage

Creating a Workflow

Vantage

3. Configure the Connection Information panel with your settings (Generic S3 shown):

Note: For details about configuring fields, click the M at the top of the connection

panel to open the man page describing the action and its fields.

For some connection types, certain fields are linked together so that as you enter a URL,

for example, the matching path is filled in as well.

You can manually enter an existing path, or you can enter a path with new folders—the

workflow will create them when you press Next to log in to the store.

If your storage allows it, when you enter the URL, you can include other data, such as a

timeout value (if needed) directly in the URL field. See the documentation for your

cloud storage account for details about possible entries in particular fields.

Click Next to connect to your cloud storage location. If the system is unavailable or you

have mis-configured the Watch action, Vantage displays this error:

Correct the problem and retry.

Using Cloud Storage with Vantage

Creating a Workflow

Vantage

When Vantage successfully connects to your cloud storage, you can view and browse

your folders (buckets or blobs):

Select the target bucket or bucket/folder path to monitor. Notice that the watch folder

path is updated at the bottom of the panel—this path is included automatically in the

URL.

Click Next to configure your monitoring settings for this action, and optionally, logging

options. For details, display the man page (M icon at the top of the panel).

When you’re done configuring the Watch action, click Finish to save your settings and

close the inspector.

Including a Copy Action to Localize the File

Configure a Copy action following the Watch to copy the Original file from its cloud

storage location to a local folder accessible to the subsequent Flip64 action.

Configuring the Flip64 Action

The Flip64 (or any other transcode) action can be configured to transcode the input file

into a specific format, and create a new video file. (How you configure the transcoding

action is irrelevant to this example workflow.)

Configure the Flip64 action to meet your transcoding requirements. When you’re done,

click Finish to save the action and close the inspector.

Using Cloud Storage with Vantage

Creating a Workflow

Vantage

Configuring the Deploy Action

The Deploy action should be configured to deliver the output file to a specified bucket

in your cloud store.

Note: You can also use other Transport actions, including Copy and Move to deliver

files, as well as Delete to remove files on your cloud storage system, based on your

workflow requirements.

To replicate the encoded output file to a specific bucket, open the inspector and

configure these Deploy action settings, for example:

1. On the Select Deployment Type panel, select Export, and then press Next.

Using Cloud Storage with Vantage

Creating a Workflow

Vantage

2. On the Select Files/File System panel:

– Check the files to deploy (listed by nickname)

– Select the file system, such as Generic S3. As the figure shows, you can choose

Azure, Aspera, FTP, Google Cloud, S3, or a number of other storage/file systems.

– Click Next to move to the next panel.

3. Configure settings in the Connection Information panel (same as the Watch action):

Note: For details about configuring fields, click the M at the top of the connection

panel to open the man page describing the action and its fields.

Using Cloud Storage with Vantage

Creating a Workflow

Vantage

For some connection types, the fields are linked together so that as you enter a URL, for

example, the matching path is filled in as well.

If your storage allows it, when you enter the URL, you can include other data directly in

the URL field. See the documentation for your cloud storage account for details about

possible entries in particular fields.

Click Next to connect to your cloud storage. If the system is unavailable or you have

mis-configured the Watch action, Vantage displays this error:

Correct the problem and retry.

When Vantage successfully connects to your cloud storage, you can view and browse

your folders (buckets or blobs):

Select the target cloud storage folder to monitor. Notice that the watch folder path is

updated at the bottom of the panel—this path is included automatically in the URL.

Click Next to set up file collision resolution in the situation where you are attempting to

deliver a file to a directory where the same file already exists. For details, display the

man page—click the M icon in the upper right corner of the panel.

When you’re done configuring the Deploy action, click Finish to save your settings and

close the inspector.

Using Cloud Storage with Vantage

Vantage

reproduced, transmitted, transcribed, altered, or translated into any languages without

written permission of Telestream, Inc. Information and specifications in this document

are subject to change without notice and do not represent a commitment on the part

of Telestream.

Telestream, CaptionMaker, Episode, Flip4Mac, FlipFactory, Flip Player, Gameshow,

GraphicsFactory, Lightspeed, MetaFlip, Post Producer, ScreenFlow, Split-and-Stitch,

Switch, Tempo, TrafficManager, Vantage, VOD Producer and Wirecast, are registered

trademarks and Cricket, e-Captioning, iQ, iVMS, iVMS ASM, Inspector, MacCaption,

Pipeline, Vantage Cloud Port, Vidchecker, and Surveyor are trademarks of Telestream,

LLC. All other trademarks are the property of their respective owners.

All other brand, product, and company names are the property of their respective

owners and are used only for identification purposes.