Today at Strata + Hadoop World, Microsoft is announcing the preview of Apache Storm clusters on Azure HDInsight. Apache Storm is an open source project in the Hadoop ecosystem which gives users access to an event-processing analytics platform that can reliably process millions of events. Now, users of Hadoop can gain insights to events as they happen in real-time.
As a managed cluster integrated into Azure HDInsight, Storm will have the benefit of being easy-to-setup (within a few clicks and a few minutes), having high availability (clusters are monitored 24/7 and under the Azure SLA for uptime), having elastic scale (where more resources can be added depending on need), and being integrated to the broad Azure ecosystem (ie. Event Hubs, HBase, VNet, etc). In many cases, Storm will integrate with an event queuing system like Event Hubs or Apache Kafka.
To try Storm, we invite you to begin by reading the following documentation:
- Overview of Storm in Azure HDInsight
- Getting started with a word count topology on Storm in HDInsight
- Develop a stream data processing application with SCP.NET and C# on Storm
- Analyze real-time sensor data using Storm and HDInsight
Additionally, at Strata + Hadoop World, our partner Hortonworks announced the future GA of Hortonworks Data Platform 2.2 as well as being Azure Certified. A key part of HDP 2.2 is the ability for Hortonworks customers to automatically move their on-premises data into Azure using Falcon. Falcon is an open-source project (found in HDP) that enables the movement and processing of datasets. With HDP 2.2, customers can schedule an automatic backup of their data in Azure. This opens up different scenarios like having a hybrid dev/test architecture, doing on-demand bursting by spinning up larger Hadoop clusters in the cloud as needed, and integration with Microsoft’s broader cloud data platform like Azure Machine Learning or Power BI for Office 365.
Read more about Microsoft’s announcements at Strata by going to the Official Microsoft Blog.