Last week Microsoft released the Azure SDK 2.2 and Service Bus SDK 2.2. These SDKs come with a new Service Bus feature: partitioned entities. Using these SDKs (or specifying api-version=2013-10 in your HTTP requests) allows you to create and use partitioned queues and topics on Azure Service Bus. Partitioned queues and topics give you improved reliability. At the same time, you should see higher maximum message throughput in most use cases.
What are partitioned queues and topics?
Whereas a conventional queue or topic is handled by a single message broker and stored in one messaging store, a partitioned queue or topic is handled by multiple message brokers and stored in multiple messaging stores. This means that the overall throughput of a partitioned queue or topic is no longer limited by the performance of a single message broker or messaging store. In addition, a temporary outage of a messaging store does not render a partitioned queue or topic unavailable.
In a nutshell, a partitioned queue or topic works as follows: Each partitioned queue or topic consists of multiple fragments. Each fragment is stored in a different messaging store and handled by a different message broker. When a message is sent to a partitioned queue or topic, Service Bus assigns the message to one of the fragments. The selection is done randomly by Service Bus or by a partition key that can be specified by the sender. When a client wants to receive a message from a partitioned queue or a subscription of a partitioned topic, Service Bus checks all fragments for messages. If it finds any, it picks one and passes that message to the receiver.
Enabling Partitioning
You can choose between three ways to create a partitioned queue or topic. The first method is to create the queue or topic from your application. Enable partitioning by setting the QueueDescription.EnablePartitioning or TopicDescription.EnablePartitioning property to true. These flags must be set at the time the queue or topic is created. It is not possible to change this property on an existing queue or topic.
Alternatively, you can create a partitioned queue or topic in Visual Studio. We added a new checkbox Enable Partitioning in the New Queue and New Topic dialog window.
The third option is the Windows Azure portal. This functionality will become available with the next portal update that is schedule to rollout at the end of this month.
Note that Azure Service Bus currently allows only 100 partitioned queues or topics per namespace. Furthermore, Partitioned queues and topics are only supported in Azure Service Bus but are not available on Service Bus for Windows Server 1.1.
Use of Partition Keys
When a message is enqueued into a partitioned queue or topic, Service Bus checks for the presence of a partition key. If it finds one it selects the fragment based on that key. If it doesn’t, it selects the fragment based on an internal algorithm.
Using a partition key
Some scenarios, such as sessions or transactions, require messages to be stored in a certain fragment. All of these scenarios require the use of a partition key. All messages that use the same partition key are assigned to the same fragment.
Depending on the scenario, different message properties are used as a partition key:
SessionId. If a message has the SessionId property set, then Service Bus uses the SessionId property as the partition key. This way, all messages that belong to the same session are assigned to the same fragment and handled by the same message broker. This allows Service Bus to guarantee message ordering as well as the consistency of session states.
PartitionKey. If a message has the PartitionKey property set but not the SessionId property, then Service Bus uses the PartitionKey property as the partition key. Use the PartitionKey property to send non-sessionful transactional messages. The partition key ensures that all messages that are sent within a transaction are handled by the same messaging broker.
MessageId. If the queue or topic has the RequiresDuplicationDetection property set to true, then the MessageId property serves as the partition key if the SessionId or a PartitionKey properties are not set. This ensures that all copies of the same message are handled by the same message broker and, thus, allows Service Bus to detect and eliminate duplicate messages
Not using a partition key
In the absence of a partition key, Service Bus distributes messages in a round robin manner to all the fragments of the partitioned queue or topic. If the chosen fragment is not available, Service Bus assigns the message to a different fragment. This way, the send operation succeeds despite the temporary unavailability of a messaging store.
As you can see, a partition key pins a message to a specific fragment. If the messaging store that holds this fragment is unavailable, Service Bus returns an error. In the absence of a partition key, Service Bus is able to pick a different fragment and the operation succeeds. Therefore, it is recommended that you do not supply a partition key unless required.
Using Transactions with Partitioned Entities
Messages that are sent as part of a transaction must specify a partition key. This can be a SessionId, PartitionKey, or MessageId. All messages that are sent as part of the same transaction must specify the same partition key.
You can use the following code to send a transactional message to a partitioned queue:
If you want to send a transactional message to a session-aware topic or queue, the message must have the SessionId property set. As stated above, the SessionId serves as the partition key. If you set PartitionKey property as well, it must be identical to the SessionId property.
Unlike with regular queues or topics, it is not possible to use a single transaction to send multiple messages to different sessions.
Additional Information
Find more detail about the partitioned queues and topics at https://msdn.microsoft.com/en-us/library/dn520246.aspx and check out the Service Bus Partitioned Queue sample.