Announcing the 0.4.0-beta preview of Microsoft Azure WebJobs SDK

We are releasing another preview of the Microsoft Azure WebJobs SDK introduced by Scott Hanselman. To read more about the previous preview, read this anouncement post.

This release has same general feature set as 0.3.0-beta and also new features.

Download this release

You can download the WebJobs SDK from the NuGet gallery. You can install or update these packages through NuGet gallery using the NuGet Package Manager Console, like this:

Install-Package Microsoft.Azure.WebJobs –Pre

If you want to use Microsoft Azure Service Bus triggers, install the following package:

Install-Package Microsoft.Azure.WebJobs.ServiceBus -Pre

Since the package names have changed from 0.3.0-beta, we have uploaded redirection packages which will help you update to the latest version.

Update-Package Microsoft.Azure.Jobs.Core –Pre

Update-Package Microsoft.Azure.Jobs –Pre

What is the WebJobs SDK?

The WebJobs feature of Microsoft Azure Web Sites provides an easy way for you to run programs such as services or background tasks in a Web Site. You can upload and run an executable file such as an .exe, .cmd, or .bat file to your web site. You can run these as triggered or continuous WebJobs. Without the WebJobs SDK, connecting and running background task requires a lot of complex programming. The SDK provides a framework that lets you write a minimum amount of code to get common tasks done.

The WebJobs SDK has a binding and trigger system which works with Microsoft Azure Storage Blobs, Queues and Tables as well as Service Bus. The binding system makes it easy to write code that reads or writes Microsoft Azure Storage objects. The trigger system calls a function in your code whenever any new data is received in a queue or blob.

Updates in this preview

Async support

You can use async/await in your functions and your functions can return Task.

Distinct functions within a single JobHost are executed in parallel. This means that if you have 2 functions listening on different Queues, then they would be executed in parallel.

The following code shows how you can use async/await and return Task in your function. This function will trigger on a new message on an Azure Queue called inputqueue and will write the message to a Blob.

class Program
{
    static void Main()
    {
        JobHost host = new JobHost();
        host.RunAndBlock();
    }
   public async static Task HelloWorldFunctionAsync(
   [QueueTrigger("inputqueue")] string inputText,
   [Blob("output/output.txt")] TextWriter output)
   {
       await output.WriteAsync(inputText);
   }
}

You can optionally take in the CancellationToken as an argument to your functions. For eg. the following function is triggered when a new Blob is detected in the container called “input”. You can pass in the CancellationToken to CopyToAsync function. This function also shows how the SDK binds the file name and extension and gives you an easy access to them.

class HelloWorldAsyncCancellationToken
{
    static void Main()
    {
        JobHost host = new JobHost();
        host.RunAndBlock();
    }
    public async static Task ProcessBlob(
    [BlobTrigger("input/{name}.{extension}")] Stream input,
    string name, // The SDK binds the name of the File
    string extension, // The SDK binds the extension of the File
    [Blob("output/{name}.{extension}", FileAccess.Write)] Stream output,
    CancellationToken token)
    {
        await input.CopyToAsync(output, 4096, token);
    }
}

You can invoke functions explicitly. This is useful if you are running a WebJob on a schedule using the Azure Scheduler or you just want to invoke functions. This approach still gives you the benefits of the SDK around diagnostics, cancelling long running functions etc.

class Program
{
    static void Main()
    {   
        JobHost host = new JobHost();
        Task callTask = host.CallAsync(typeof(Program).GetMethod("ManualTrigger"), 
                                      new { value = 20 });

        Console.WriteLine("Waiting for async operation...");
        callTask.Wait();
        Console.WriteLine("Task completed: " + callTask.Status);
    }

    [NoAutomaticTrigger]
    public static void ManualTrigger(
    TextWriter log, int value, [Queue("outputqueue")] out string message)
    {
        log.WriteLine("Function is invoked with value={0}", value);
        message = value.ToString();
        log.WriteLine("Following message will be written on the Queue={0}", message);
    }
}

Handling Poison messages in Azure Queues

In 0.3.0-beta the SDK gave you an option of binding to the DequeueCount  property of Queues and now, in this release we are adding support for automatically moving the message to a poison queue.

You can now process poison messages in your application code, such as logging them for investigation. Just bind a function to QueueTrigger(“queuename-poison”).

The following code will process the queue message. When a function is bounded to a Queue and an exception happens while processing the function, the SDK will process the message 5 times(default) before marking the message as being poisoned and the SDK will move the message to a different queue.

class ProcessPoisonMessages
{
    static void Main()
    {
        JobHost host = new JobHost();
        host.RunAndBlock();
     }
     public async static Task ProcessQueue(
     [QueueTrigger("inputqueue")] string inputText,
     [Blob("output/output.txt")] TextWriter output)
     {
       await output.WriteAsync(inputText);
     }
     public static void ProcessPosionQueue(
     [QueueTrigger("inputqueue-poison")] string inputText)
     {
       //Process the poison message and log it or send a notification
     }
}

Better polling logic for Azure Queues

This release has a new polling strategy. The SDK now implements a random exponential back-off algorithm to reduce the effect of idle queue polling on storage transaction costs.

 

Fast Path notifications for Queues

• In 0.3.0-Beta, the SDK used to poll every ~2 sec. This meant that if your app had a chain of 20 functions (a function was writing to a Queue, triggering another function which would then write to another queue and trigger another function and so on) , then it would have taken ~40 sec to process the 20 queue messages. With these changes this now takes ~8 sec.

 

Configuration options for queue polling

The SDK exposes a few knobs where you can configure the Queue polling behavior.

• MaxPollingInterval for when a queue remains empty, the longest period of time to wait before checking for a message to. Default is 10min.
• MaxDequeueCount for when the Queue message is moved to a poison queue. Default is 5

 

The following code shows how to configure these settings:

static void Main()
{
       JobHostConfiguration config = new JobHostConfiguration();
       config.Queues.MaxDequeueCount = 3;
       config.Queues.MaxPollingInterval = TimeSpan.FromMinutes(20); 
       JobHost host = new JobHost(config);
       host.RunAndBlock();
 }

 

Package/ Namespace changes

We are changing the package name to avoid ambiguity with the generic term Job which can be confusing and sometimes difficult to search for.

You will have to recompile your existing apps and change ConnectionStrings to incorporate these changes.

Asset	Before	After
Packages	Microsoft.Azure.Jobs	Microsoft.Azure.WebJobs
	Microsoft.Azure.Jobs.Core	Microsoft.Azure.WebJobs.Core
	Microsoft.Azure.Jobs.ServiceBus	Microsoft.Azure.WebJobs.ServiceBus
Namespaces	Microsoft.Azure.Jobs	Microsoft.Azure.WebJobs
ConnectionString Names	AzureJobsStorage	AzureWebJobsStorage
	AzureJobsDashboard	AzureWebJobsDashboard

 

Faster dashboard index processing

Improve performance of the dashboard when it shows all the WebJobs and function details for a WebJob

Dashboard data out of date warning

The dashboard now processes host data in the background and shows a warning if there’s a lot of work left to do.

Dashboard indexing errors

In the “About” page, the dashboard shows indexing errors if any. These are useful since if the dashboard fails to index any logs you can go and check this page to find out if there were any errors

Bug Fixes

This release has lots of bug fixes. We did prioritize the bugs reported on the forums and stackoverflow.

 

Existing features of the SDK

Following is the feature set that was supported in 0.3.0-beta and continues to be supported in this release.

Azure usage

The SDK adds Triggers and Bindings for Azure Blobs, Queues, Tables and ServiceBus.

Triggers

Functions get executed when a new input is detected on a Queue or a Blob. For more details on triggers please see this post.

Bindings

The SDK supports binding to provides model binding between C# primitive types and Azure storage like Blobs, Tables, and Queues. This makes it easy for a developer to read/ write from Blobs, Tables and Queues as they do not have to learn about the code around reading/ writing from Azure Storage.

The following Bindings are currently supported: Stream, TextReader/Writer, and String. You can add support for binding to your custom types and other types from the Storage SDK as well. You can look at the samples listed below for more information.

Hosting

A JobHost is an execution container which knows what functions you have in your program. A JobHost object (which lives in Microsoft.Azure.WebJobs ) reads the bindings, listens on the triggers, and invokes the functions. In the following example, you create an instance of JobHost and call RunAndBlock(), which will cause the JobHost to listen for any triggers on any functions that you define in this Host.

static void Main()
{
    JobHost host = new JobHost();
    host.RunAndBlock();
}

Dashboard for monitoring WebJobs.

As WebJobs (written in any language and of any type) execute, you can monitor them in real time. You can see their state (Running, Stopped, Successfully completed), last run time and the logs of a particular execution. The following screenshot shows you a view of all WebJobs running in your Website.

Function execution details

When you are monitoring a particular execution of this “ImageProcessing” WebJob, you can view invocation details about the functions in the program such as:

• The parameters of this function.
• How much time it took for the function to execute.
• How much time it took to read from a Blob and how many bytes were read/ written.

Following is the code for an ImageProcessing WebJob. This WebJob uses the

using Microsoft.Azure.WebJobs;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using System.Web.Helpers;

namespace ImageResizeAndWaterMark
{
    class ImageProcessingFunctions
    {
        public static void Resize(
        [BlobTrigger(@"images-input/{name}")] WebImage input,
        [Blob(@"images2-output/{name}")] out WebImage output)
        {
            var width = 80;
            var height = 80;
            output = input.Resize(width, height);
        }
        public static void WaterMark(
        [BlobTrigger(@"images2-output/{name}")] WebImage input,
        [Blob(@"images2-newoutput/{name}")] out WebImage output)
        {
            output = input.AddTextWatermark("WebJobs is now awesome!!!!", fontSize: 6);
        }
    }
    public class WebImageBinder : ICloudBlobStreamBinder
    {  
        public Task ReadFromStreamAsync(Stream input, CancellationToken cancellationToken)
        {
            return Task.FromResult(new WebImage(input));
        }

        public async Task WriteToStreamAsync(WebImage result, Stream output, CancellationToken cancellationToken)
        {
            var bytes = result.GetBytes();
            await output.WriteAsync(bytes, 0, bytes.Length,cancellationToken);
        }
    }
}

Invoke & Replay

In the above example if the Resize function fails for some reason, you can upload a new image and Replay Resize function, which will trigger the execution chain and call Watermark function as well. This is useful to diagnose and debug an issue when you have a complicated graph for chaining functions together. You can also run a function from the dashboard.

Causality of functions

In the above example, we know that when the Resize function writes to a Blob, it will trigger the WaterMark function. The dashboard will show this causality between functions. If you have chained lots of functions which will get triggered as new inputs are detected then it can be useful to see this causality graph.

Search Blobs

You can click on Search for a Blob and get information on what happened to that Blob. For example, in the case of the ImageProcessing, the Blob was written because the Resize function got executed. For more details on Search Blobs see this post.

Samples

Samples for WebJobs SDK can be found at https://github.com/Azure/azure-webjobs-sdk-samples

• You can find samples on how to use triggers and bindings for Blobs, Tables, Queues and Service Bus.
• There is a sample called PhluffyShuffy which is an Image processing Website where a customer can upload pictures which will trigger a function to process those pictures from Blob storage.

Documentation

• Tutorial: Getting Started with the Windows Azure WebJobs SDK
• Channel 9 video on Making Your Jobs Easier With Windows Azure WebJobs SDK
• Introduction to WebJobs and SDK by Scott Hanselman
• A web site monitor built using WebJobs and ASP.NET by Brady Gaster
• Azure WebJobs – Recommended Resources
• WebJobs video series on Azure Friday

Deploying WebJobs with SDK

Visual Studio 2013 Update3/ Azure SDK 2.4 added Visual Studio Tooling support to publish WebJobs to Azure Websites.For more information, see How to Deploy Azure WebJobs to Azure Websites

Known Issues when migrating from 0.3.0-beta to 0.4.0-beta

Update namespaces to match new APIs

Before
Microsoft.WindowsAzure.Jobs
Microsoft.Azure.Jobs.Core

After

Microsoft.WindowsAzure.WebJobs
Microsoft.Azure.WebJobs.Core

Update connectionString names

When you are setting the connectionStrings, in the app.config of your WebJob or in “Configure Tab” in your Microsoft Azure Web sites, you will have to change the connectionString names as follows

Before

AzureJobsRuntime
AzureJobsDashboard

After

AzureWebJobsRuntime
AzureWebJobsDashboard

Storing connectionString in the logging account

Until the latest version of the SDK (0.4.0-beta), your AzureJobsRuntime storage account connection string was stored in the AzureJobsDashboard logs. As only administrative users can view the logs and those same users can view and reset the connection strings we feel this hasn’t been a high impact problem however going forward we will stop this practice. Over the next month we will update the dashboard to remove any previously stored connection strings.

To remove these now please open the logs specified by AzureJobsDashboard connectionString.

Delete the following Blob container names with name as “azure-jobs*” except for the following  “azure-jobs-host”, “azure-jobs-host-output”, “azure-jobs-host-archive” and “azure-jobs-dashboard”

Delete the tables with name as “AzureJobs*”

Delete the queues with the name as “azure-jobs*”

Dashboard will only work for WebJobs deployed with 0.4.0-beta

If you had a WebJob deployed with 0.3.0-beta version of SDK and, if you access the dashboard to see the logs for the WebJob, then you will see a warning about “Host not running”. This happens because as part of this release a newer version of the dashboard gets deployed to all Azure Websites. The new dashboard has some protocol changes which are not compatible with 0.3.0-beta. To work around this error, please update your WebJob to use 0.4.0-beta NuGet package and redeploy your WebJob.

Give feedback and get help

The WebJobs feature of Microsoft Azure Web Sites and the Microsoft Azure WebJobs SDK are in preview. Any feedback to improve this experience is always welcome.

If you have questions that are not directly related to the tutorial, you can post them to the Azure forum, the ASP.NET forum, or StackOverflow.com. Use #AzureWebJobs SDK for Twitter and the tag Azure-WebJobsSDK for StackOverflow.