Storage for Media Services

Storage is a very critical component of Media workflows. When you create a Media Services account, you are also asked to create or select an existing Storage account.  Media Services uses the concept of Assets to manage your media content.  The media files for an Asset are stored in your storage account but entity metadata for the Asset is stored by Media Services in its internal repository. This blog covers the following topics that you should pay attention to when using Storage with Media Services.

• Storage Redundancy
• Storage Keys
• Scaling Storage
• Storage Analytics and Logging
• Media Assets in Storage
• Uploading Assets
• Storage Cost

 

Storage Redundancy

The data in your Microsoft Azure storage account is always replicated to ensure durability and high availability, meeting the Azure Storage SLA even in the face of transient hardware failures. Azure Storage offers a few options for redundancy. For the Storage accounts associated with your production Media Services account, it’s best to set your storage account as “Geo-redundant storage (GRS)” or as “Read-Access Geo-redundant storage (RA-GRS)” as these options offer you protection against region level disasters such as floods, hurricanes etc. You can change how your data is replicated after your storage account has been created, but note that you may incur an additional one-time data transfer cost if you switch from LRS (Locally Redundant Storage) to GRS or RA-GRS. More information on how you can manage storage account replication can be found on How to: Manage Storage Account Replication article. Note that there is different pricing for LRS, GRS and RA-GRS. Details can be found on Storage Pricing Details page  

Storage Keys

When you create a storage account, Azure generates two 512-bit storage access keys, which are used for authentication when the storage account is accessed. It is important that you keep these keys secured. Several compliance standards require periodic key regeneration and depending on what compliance standards your organization is targeting you will need to define a policy for key regeneration. Media Services needs one storage key to read and write media data to your storage account. If you create a Media Services account via the Azure Management Portal, the primary storage key is the one that gets stored with Media Services. If you regenerate the storage key(s), you must provide a valid updated key so that your Media account continues to function as expected. You can regenerate the keys one at a time to avoid disruption of your Media Services based application The article on How To – Update Media Services after rolling storage access keys describes that.

Scaling Storage

A single storage account has certain scale and performance limits which are well documented in Azure Storage Scalability and Performance Targets. Since media files are huge, a single storage account may not be enough for your business. Media Services provides the ability to attach multiple storage accounts with a single Media Services account. Attaching multiple storage accounts to a Media Services account provides the following benefits:

• Load balancing your assets across multiple storage accounts.
• Scaling Media Services for large amounts of content storage and processing.
• Isolating your mezzanine (source) file storage from your streaming or DRM protected file storage.

For more information, see the following topicssamples: Managing Media Services Assets across Multiple Storage Accounts Managing assets across multiple storage accounts In Azure Media Services and defining load balancing strategy How to: Manage Multiple Storage Accounts Attached to a Media Services Account Note that currently, the only way to attach multiple storage accounts is by using Azure Service Management REST API  

Storage Analytics and Logging

Azure Storage Analytics performs logging and provides metrics data for a storage account. You can use this data to trace requests, analyze usage trends, and diagnose issues with your storage account. You can enable Storage Analytics from the Azure Management Portal; for details see How To Monitor a Storage Account. Media Services uses Blob Service to store your media data and hence it is a good idea to turn on Verbose monitoring for Blobs and also configure logging for Blobs with a retention policy of a few days. The Storage service stores the aggregated analytics data in well known Blobs and Tables, which can be accessed using the Blob service and Table service APIs respectively. See Storage Analytics for more information on this topic. Specifically pay attention to the following topics: Metrics table schema and Log format. For Media Services, Storage is involved in ingest, encode as well as delivery workflows. Understanding the metrics table scheme and logging format will help you a lot in analyzing failures that your application encounters. Also beginning from Storage Client 2.1 release, you can enable client side tracing regarding request execution and REST requests. Additionally, Azure Diagnostics provides a trace listener that can redirect client trace messages to the WADLogsTable if you wish to persist these traces in the cloud. See the “Client Tracing” section in the blog announcing Storage Client 2.1 release to learn more about log data format. Client side tracing used in conjunction with storage logging will give you a complete view of your applications interactions with storage

Additional Resources related to Storage Analytics

• Storage Analytics Video – A short 5 and a half minute video showing how to enable and use storage analytics.
• Storage Analytics SDP (Support Diagnostics Platform) Package – Allows you to quickly and easily gather all the storage analytics logs
• Storage Analytics Billing – MSDN topic discussing the cost of Storage Analytics and how to use metrics and logging data to understand your storage services bill
• Tracking Storage Requests – Blog covering the topic of how to use logs to track storage requests
• AzCopy – A tool for uploading/downloading files for Azure Storage Blobs (can be used to download logs)

 

 

Media Assets in Storage

As mentioned earlier, an Asset is a logical unit that represents a single audio visual presentation.  An Asset contains a collection of one to many media files represented by an entity called AssetFile. Media Services uses a container in Storage to represent an Asset. The AssetFiles are stored as Block Blobs in the Asset container. There are several Storage Explorer applications that are available that you can use to view data in Storage. See Azure Storage Explorers to learn more about these applications. Please do not add, edit or delete the asset containers or blobs within the container outside of the Media Services APIs. Doing so may cause issues when trying to access the Asset in your media application built using Media Services APIs. Also note that currently block blobs have an upper limit of 200 GB and hence an AssetFile cannot be larger than 200 GB  

Uploading Assets

Media files are huge and depending on your network connectivity to Azure datacenters, you may want to consider using a UDP based transport technology such as the one provided by Aspera to upload media content in your Storage account. Aspera provides a service called “Aspera On Demand for Microsoft Azure” that provides access to Aspera’s fasp high-speed transfer software on Azure. You can purchase a subscription for this service from Azure Store. See Aspera on Demand for Microsoft Azure to learn more about Aspera’s offering on Azure. Also look at Aspera On Demand for Microsoft Azure FAQ to get answers to some common questions. You can use Aspera’s service in conjunction with Media Services Bulk Ingest APIs to ingest your catalog of media content in bulk. See Ingesting Assets in Bulk with Media Services SDK for .NET to learn more about this. Note that you can use the Bulk Ingest REST API if you do not want to program in .NET For additional information on this topic, see Upload Media topic in MSDN  

Storage Cost

The Storage Pricing Details page provides information about how much storage would cost you. With Media Services, your Azure account will be charged for Block Blobs (As mentioned earlier, each asset is represented as a container and the AssetFiles are stored as Block Blobs in the container). So if you create an Asset and add one AssetFile that is 1 GB in size, you will be charged for 1 GB of Block Blob storage. The cost will vary depending on whether you setup your storage account as LRS, GRS or RA-GRS. In addition to the Block Blobs, Storage Transaction costs will apply. Also Data Transfer charges will apply if you transfer data external to the region in which your Storage account resides. Note that you do not get charged for data transfer within an Azure datacenter but data transfer charges do apply if you transfer data to an Azure service in a different Azure region. Data Transfer within a datacenter will occur when Media Services executes encoding jobs as data is copied from your storage account to the actual VM where the encoding operation will take place.