• <1 minute

Sub-Clipping and Live Archive Extraction with Media Encoder Standard

Perform editing operations on your live stream such as generating frame-accurate highlights, archiving to a MP4 file or using Media Encoder Standard.

Nearly every customer doing live streaming has asked me for the ability to edit the live streams, either during or after the event. I get asked about creating sub-clips, and creating ISO MP4 archives of the entire event for subsequent syndication.

Today, I am pleased to announce you can now use Media Encoder Standard to perform such editing operations on your live stream. In this blog post, I will show you how easily you can achieve this using Azure Media Services Explorer (AMSE).

Getting Started

To get started, follow the below steps:

  1. Install the latest (v3.28 or newer) Azure Media Services Explorer (AMSE) tool.
  2. Use your Azure Media Service account to create a Channel, and associated Program(s) for a live event. See overview of live streaming for tips.
  3. Get familiar with the rendered sub-clips concept.

Also, I recommend that you start with Programs of moderate durations (tens of minutes to a couple of hours) to get familiar with the tool and features.

Archiving the top bitrate

This is the easiest scenario, where you wish to take the highest bitrate video and highest bitrate audio and produce a single bitrate ISO MP4 file for the full Program. The steps are:

  1. Launch AMSE.
  2. In the Assets tab, browse to the Asset that represents your Program.
  3. Select that Asset and hit Alt+R.
  4. The “Live stream/archive Subclipping” dialog will pop up.
  5. Select the default option, “Archive Top Bitrate” and hit “Subclip.”
  6. AMSE will submit an encode Task to Media Encoder Standard with the appropriate configuration.

Archive Top Bitrate

In AMSE, if you go to the Jobs tab, select the submitted Job, and double-click on it, the Job Information window will open. Go to the Tasks tab, and select Configuration to look at the XML configuration for this encode Task:

Task Configuration XML for Archiving Top Bitrate

Note how the VideoStream as well as AudioStream elements are set to a special value, “TopBItrate”, telling the encoder to pick the highest quality video and audio to process. In the Encoding section, the encoder is told to copy the source video and audio (CopyVideo and CopyAudio elements). Consequently, the encoder simply reads all the fragments of video and audio, repackages them into an audio-video interleaved ISO MP4 file.

When the job completes, the output will contain an ISO MP4 file.

Trimming the top bitrate at GOP/fragment boundaries

This scenario is an extension of the previous one, where you wish to take a portion of the highest bitrate video and highest bitrate audio and produce a single bitrate ISO MP4 file for the  selected sub-clip. As previously explained, if the in/out points of your edits line up with GOP or fragment boundaries, then all that Media Encoder Standard needs to do is to repackage the archived fragments into an audio-video interleaved ISO MP4 file. The steps are:

  1. Launch AMSE.
  2. In the Assets tab, browse to the Asset that represents your Program.
  3. Select that Asset and hit Alt+R.
  4. The “Live stream/archive Subclipping” dialog will pop up.
  5. Select the default option, “Archive Top Bitrate” and check the “Trim the live archive/stream (GOP accurate)” box. Since playback the Asset is selected by default, AMSE will use Azure Media Player to playback the live stream in the window.
  6. Use the playback controls to identify the approximate start/end of the portion of interest.
  7. Then move the sliders for “Start time” and “End time” to positions on the timeline so that the Start time is at or before the beginning of the clip of interest, and End time is at or after the end of the clip of interest. Note how the values for Start and end time increment by units of fragment duration. In the screenshot below, the first frame of interest begins at 2min 24sec into the video, but the Start time for the sub-clip Task is set to 2min 23.10 sec.
  8. Once you have adjusted the sliders as desired, hit “Subclip” to kick off the encode Task. When the job completes, the output Asset will contain an ISO MP4 file corresponding to the sub-clip.

Sub-Clip Top Bitrate

If you open the Job information page in AMSE and look at the Task configuration XML, you will note that AMSE has added StartTime and Duration attributes to the Source element:

Source StartTime=”P16682DT17H23M39.0666668S” Duration=”PT25.758S”.

AMSE takes care of the conversion of the in/out points of the sub-clip into the appropriate values. If you need more information about the details, please look into the source code for AMSE.

Archiving all the bitrates

This is an extension to first section, where you may want the output on demand Asset to contain all the video bitrates and all the audio bitrates present in the input Live Asset. This option should be rarely needed; the original Asset should suffice for all your needs. This capability is implemented for the sake of completeness.

In the “Live stream/archive Subclipping” dialog box, you can select “Archive All Bitrates” option before submitting the encode Task. If you bring up the Task configuration XML, you will see that the VideoStream as well as AudioStream elements are set to a special value, “*”, telling the encoder to process all bitrates.

After the task completes, the output will contain a set of ISO MP4 files. Each video bitrate in the live stream will be interleaved with all the audio bitrates to produce each MP4 file. The output Asset will also contain a so-called manifest file (*.ISM) for using with Dynamic Packaging.

Trimming the Live Asset at GOP/fragment boundaries

This scenario is an extension of the second scenario, where you wish to extract a portion of the live stream and create a new on-demand Asset that can be streamed. As previously explained, the benefits to this method are that your sub-clips can have their own life-cycle independent from the stream they were cut from, and can have their own security properties. For instance the full event may be access-controlled but the sub-clips can be in the clear.

The steps to perform this are straightforward: Select the “Archive All Bitrates” option, then check the “Trim the live archive/stream (GOP accurate)” box to activate the start and end time controls as described in the second section. As in the previous scenario, after the task completes, the output Asset will contain a set of ISO MP4 files along with manifest (*.ISM) file.

Frame-Accurate Rendered Sub-Clips

In all the previous scenarios, the in/out points of the sub-clipping operation has been restricted to GOP or fragment boundaries. If you instead need a frame-accurate edit of a portion of the live stream, then you would use the so-called Rendered Sub-Clip feature. Here, the encoder will take the top bitrate streams and re-encode (“render”) them to specified bitrates and resolutions.

To use this feature, bring up the “Live stream/archive Subclipping” dialog as explained above, and now select the “Reencode Top Bitrate” option, check the “Trim the live archive/stream (GOP accurate)” box to activate the start and end time controls. Naturally, since you are performing frame-accurate operations, you can edit the Start and End times freely.

Note: As of version 2.38 release of AMSE, it is not possible to step through the live stream frame-by-frame to specify in and out points. We are working on other tools that can support this operation. If you have a need for such tools, please reach out to us at amslived@microsoft.com

Rendered Sub-Clip Selection

And now, when you hit the “Subclip…” button, AMSE opens up the regular Media Encoder Standard dialog box for specifying the encode settings. You can select the “H264 Multiple Bitrate 720p” preset, as an example, and update the values as needed. For more information on the presets, see our documentation.

Encode Options for Rendered Sub-Clip

As in the previous scenario, after the task completes, the output Asset will contain a set of ISO MP4 files along with manifest (*.ISM) file.

Known Issues

Following are some of the known issues with this feature.

  1. Media Encoder Standard assumes that the live stream is ‘clean.’ That is, all the fragments have been received and archived. If some fragments were dropped, for example due to a network outage, the encoder does not do a good job of processing the gaps. The output video may have video/audio misalignment.
  2. In the current release, it is not possible to select particular audio tracks. For example, if you are sending in a live stream with multiple language audio tracks, then any sub-clipping Task will end up processing all the audio tracks.
  3. Note that there is a lag between when the Task is submitted, and when Media Encoder Standard actually starts processing this Task. This lag is variable. The timestamps in the Task configuration have to be ‘valid’ when the encoder is doing the work – you may need to account for timestamps falling out of the DVR window if you are using this feature on a ‘live’ Asset.