This is the Trace Id: 1114b418e170bea5e70533be75aa9d26
Skip to main content
Azure

Azure HDInsight

Provision cloud Hadoop, Spark and HBase clusters.
Try Azure for free Create a pay-as-you-go account
A woman sitting at a desk with a computer and a plant in the background.
OVERVIEW

Manage your big data needs in an open-source platform

Run popular open‑source frameworks like Hadoop, Spark, Hive, and Kafka with Azure HDInsight—an enterprise‑grade, customizable analytics service that processes massive data at global scale and simplifies migrating big data workloads to the cloud.
  • Stay up to date with the newest releases of open source frameworks, including Kafka, HBase, and Hive LLAP. HDInsight supports the latest open-source projects from the Apache Hadoop and Spark ecosystems.
    Logos of big data tools including Spark, Hive, HBase, Kafka, Hadoop, Oozie, YARN, ZooKeeper, and Ambari.
  • Build your data lake through seamless integration with Azure data storage solutions and services including Azure Synapse Analytics, Azure Cosmos DB, Azure Data Lake Storage, Azure Blob Storage, Azure Event Hubs, and Azure Data Factory. Control costs by choosing from a wide variety of virtual machines and by leveraging load- or schedule-based autoscaling features. Monitor your entire data lake using Azure Monitor dashboards.
    Data pipeline diagram showing unstructured and structured sources flowing through Data Factory to Data Lake and HDInsight.
  • Use your preferred productivity tools, including Visual Studio, Eclipse, IntelliJ, Jupyter, and Zeppelin. Write code in familiar languages such as Scala, Python, R, JavaScript, and .NET.
    Visual Studio Code showing Hive query scripts and an HDInsight cluster list with running Spark and Hadoop services.
Features

Managed open-source clusters for secure, scalable analytics

Fast cluster setup

Open-source projects and clusters are easy to spin up quickly without the need to install hardware or manage infrastructure.

Flexible cost control

Big data clusters reduce costs through autoscaling and pricing tiers that allow you to pay for only what you use.

Trusted data protection

Enterprise-grade security and industry-leading compliance with more than 30 certifications helps protect your data.

Modern open‑source support

Optimized components for open-source technologies such as Hadoop and Spark keep you up to date.
Security

Embedded security and compliance

34,000
Full-time equivalent engineers dedicated to security initiatives at Microsoft.
Learn more
15,000
Partners with specialized security expertise.
 
Learn more
>100
Compliance certifications, including over 50 specific to global regions and countries.
Learn more
Learn more about security on Azure
A person sitting in a car using a laptop.
Pricing

Pay for only what you need

HDInsight offers a broad range of memory- or compute-optimized platforms (virtual machines). Choose the one that best suits your performance and cost requirements.
See Azure HD Insight pricing
Customer stories

Trusted by companies of all sizes

  1. Myntra logo
  2. GAP Logo
A purple background with the Microsoft logo featuring the letter M.
Myntra logo

Myntra accelerates its digital transformation

Myntra has worked closely with Microsoft to migrate its platform—from supply chain management to inventory to site capabilities to Azure for trusted, always-on, hyperscale and cost-effective computing.
A man and woman looking at a laptop.
GAP Logo

Gap Inc. accelerates its digital transformation

By building and centralizing its data platform on Azure, Gap Inc. can now apply advanced analytics and machine learning to gain a comprehensive understanding of customers across channels in all brands in its portfolio.
Back to Customer stories section
Resources

HDInsight resources and documentation

  1.
A person using a laptop.

Create Apache Spark cluster in Azure HDInsight using Azure portal

Learn how to use Apache Spark SQL with Jupyter notebooks in HDInsight.
Learn more
Two people seated on a sofa discussing content on a laptop in a living room setting with framed artwork in the background

Building Open Source Software (OSS) Analytics Solutions with Azure HDInsight

Explore training to build open-source analytical solutions on HDInsight.
Learn more
A man and woman looking at a laptop.

What is Azure HDInsight?

Learn about the full capabilities and architecture of Azure HDInsight.
Learn more
Back to RESOURCES - Learn tab section
Back to tabs
FAQ

Frequently asked questions

  • You would benefit from Azure HDInsight if you use custom code to process and analyze extremely large datasets with the latest big data processing frameworks such as Spark, Hadoop, Hive, Kafka or Hbase. Azure HDInsight gives you full control over the configuration of your clusters and the software installed on them. You might also consider HDInsight if you are migrating Hortonworks, Cloudera, or MapR clusters from on-premises environments or other clouds.
  • Azure HDInsight can be used for a variety of scenarios in big data processing. It can be historical data (data that's already collected and stored) or real-time data (data that's directly streamed from the source). The scenarios for processing such data can be summarized in the following categories: batch processing (ETL), data warehousing, Internet of Things (IoT), data science, and hybrid.
  • To learn more about HDInsight clusters types and provisioning methods, read our documentation about how to set up clusters in HDInsight with Apache Hadoop, Apache Spark, Apache Kafka, and more.
Person standing behind a laptop with arms crossed, blueprints on wall, modern workspace setting.
Next steps

Get started with an Azure free account

Pay as you go or try Azure free for up to 30 days.
Try for free
A woman in a green shirt with short curly hair smiling while looking at a woman wearing yellow shirt.
Azure Solutions

Learn about more Azure cloud solutions

Solve your business problems with proven combinations of Azure cloud services, as well as sample architectures and documentation.
Explore Azure AI solutions
A man in a white shirt using a laptop.
Business Solutions Hub

Find the right Microsoft Cloud solution

Browse the Microsoft Business Solutions Hub to find the products and solutions that can help your organization reach its goals.
Explore solutions