Secure and reliable ingestion of very large amounts of telemetry data from multiple distinct sources is the key to many modern, adaptive interactive experiences as well as for many innovative consumer and commercial products, and services.
Within the next few years, most new vehicles sold will feature driver assistance and vehicle reliability and safety capabilities uniquely enabled by insights derived from information collected across the vehicle fleets on the road. Many innovative ideas in industrial manufacturing, energy and natural resource management, health care, and municipal utilities as well as in traffic and transport control infrastructure depend on capture and correlation of data from very many different sources in near real time. More and more popular television formats engage the audience in real time and producers want to expand the engagement depth, and game studios keep pushing the envelope on making massively multiplayer experiences richer with ever growing depth.
What’s common across all these scenarios is that they present enormous challenges in terms of data capture. Either data needs to be ingested at very high sustained throughput rates, or from a very large number of concurrent clients, or both. A popular live TV show with several million viewers who all push a voting button at the same time and where a tallied-up result is expected within a few seconds is a formidable challenge. Capturing live vehicle telemetry during rush hour in Europe, the U.S. East Coast, or in the key population centers in Asia is far beyond the scale systems built by the vehicle manufacturers ever had to deal with.
Event Hubs is a new capability now commercially available in Microsoft Azure Service Bus that provides a platform-as-a-service solution for these and many other scenarios. Event Hubs can help realizing telemetry ingestion scenarios with over 1 GByte/sec cumulative throughput and from hundreds of thousands of concurrently connected clients with all the hard scale-out and reliability problems being handled by the service.
An Event Hub can be easily created either through the Azure portal or through the Management API and is instantly available with no further setup or management/maintenance requirements.
Each Event Hub can be self-service configured with between 8 and 32 partitions, which provide parallelism for concurrent event consumption into backend systems or for high throughput submission into the system. The throughput quota is governed by “throughput units”, each allowing for 1MByte/s send operations (up to 1000 operations) and 2MByte/s reads. Throughput units can be added (and removed) from the Event Hub’s namespace at any time through the portal and are billed by the hour. The self-service limits of at most 32 partitions and at most 20 throughput units can be lifted through contacting Microsoft Azure product support.
Events can be submitted directly into Event Hubs from practically any kind of networked device either using plain HTTPS requests or the ISO/IEC standard AMQP 1.0 protocol, both with an effective and yet simple security model that allows for per-sender access control while not having to manage accounts on a per-sender basis.
The consumption of events is accomplished through the AMQP 1.0 protocol, supported across a broad number of operating systems, runtimes, and programming languages, and which allows for efficient and reliable delivery with well tunable flow control. Unlike with common queueing or publish/subscribe systems, the read offset is controlled and handled by the client, allowing tight control over processing and re-processing of events over a data retention window of up to 7 days in self-service configuration and up to 30 days enabled through Azure support, and also allows for time-correlated consumption from multiple event streams.
Event Hubs explicitly support external leader-election and failover strategies for concurrent consumers as they are commonly employed in high-scale, always-on distributed systems infrastructures and provides an easy-to-use standard implementation of a scale-out aware consumer API for .NET developers for and, shortly, also for Java in form of a spout for Twitter Storm.
Microsoft Azure’s Event Hubs are the direct result of customer requests and customer feedback, and we are inviting you to try them out today, and to also let us know about your experiences with the product, and the scenarios that you are looking to enable.
Your Microsoft Azure Service Bus Team