Application Packages and Task Dependencies now available on Azure Batch

Last year we released a cloud-based job scheduling service called Azure Batch. Alongside this highly scalable service, we also previewed Batch Apps which included some higher-level features. As promised, we’ve been hard at work porting these preview features to the generally available, production-ready Batch service. Today we’re announcing two of the most popular features: Application Packages and Task Dependencies.

Application Packages

Each workload requires a common set of files like executables, libraries, and additional metadata. This set of files can be workload specific and require downloading to the VMs before running a Job or a Task. The Application Packages feature will provide easy management and deployment of these packages of files to the compute nodes in your Pool. It helps dramatically decrease the complexity of managing all the diverse files that make up an application as you’ll be able to upload and manage multiple versions, and also automatically deploy one or more of these application packages to your nodes in a Pool. Additionally, you won’t need to manage Azure Storage to move these files around, as Batch will handle all the details in the background by working with your storage account to securely store and deploy these packages.

Application packages are simply ZIP files, and can contain any number of files that are required on the compute nodes. You’ll be able to manage these applications, including linking a storage account, uploading multiple versions, deleting existing applications, and configuring default versions for each application. These actions can be easily performed through the Azure portal by navigating to your Batch Account.

We’ve made it very simple to install application packages on compute nodes, which is done by specifying one or more application package references for a pool. You can do this at Pool creation time.

// Create the unbound CloudPool
CloudPool myCloudPool =
    batchClient.PoolOperations.CreatePool(poolId: "myPool",
                                          osFamily: "4",
                                          virtualMachineSize: "small",
                                          targetDedicated: "1");

// Specify the application and version to install on the compute nodes
myCloudPool.ApplicationPackageReferences = new List
{
    new ApplicationPackageReference {
        ApplicationId = "litware",
        Version = "1.1001.2b" }
};

// Commit the pool so that it's created in the Batch service. As the nodes join
// the pool, the specified application package will be installed on each.
await myCloudPool.CommitAsync();

Once the application packages have been installed on the compute nodes in your pool, you’ll be able to execute files within your packages by using environment variables that are created on your behalf for your task command lines. This variable adheres to a naming scheme – AZ_BATCH_APP_PACKAGE_appid#version where appid refers to the application name, and the version refers to the version you want to run. You can also reference just the appid if you’ve picked a default version in the Azure Portal – AZ_BATCH_APP_PACKAGE_appid.

Task Dependencies

Many jobs and workloads require dependencies between tasks, Batch Apps supported this via Stages. We’ve now added Task Dependencies feature to Batch which be used to replicate Batch Apps-style Stages, or more generally to represent any dependency relationship between tasks. This feature helps build relationships among tasks which determine the order in which activities or work needs to be performed. Tasks can have multiple preceding tasks and multiple succeeding tasks.

You can utilize this feature include a pre-processing or rendering task that generates a point cloud, frame tasks that depend on the pre-rendered task, and even where individual frames or tasks are combined into a merge task for a final output. You can use the id of the pre-processing task to invoke the dependency.

frameTask.DependsOn = TaskDependencies.OnId("prerender"); 

You can also have tasks that depend on a range of tasks. Perhaps you have a merge task that depends on all other tasks completing first. This can be achieved by simply invoking another helper method, “OnIdRange”:

makeMovieTask.DependsOn = TaskDependencies.OnIdRange(startFrame, endFrame); 

These features are a great addition to the Batch service, and will help developers get to market faster and reduce overhead by leaving the boilerplate code and plumbing to us. These features are now live and available in the new Azure.Batch 3.1 NuGet package and corresponding features now available in the Azure portal. Additionally, you can download the sample code here.

We encourage you all to try out both Application Packages and Task Dependencies and provide us feedback directly at wabatch@microsoft.com or by posting on our official forum.

Thank you!

The Azure Batch Team