Learn what Hadoop components and versions are included in HDInsight.
This page provides a quick overview of the learning resources for Azure HDInsight. Use the diagram to guide you in the most effective learning path.
Get an overview of HDInsight components, common terminology, and scenarios. HDInsight deploys and provisions Apache Hadoop clusters in the cloud, providing a software framework designed to manage, analyze, and report on big data.
Learn how to provision clusters, query data with Hive, and output to Excel for analysis.
Learn about HBase, a NoSQL database built on Hadoop and designed for very large amounts of data. HBase clusters on HDInsight are configured to store data directly in Azure Blob storage, with low latency and increased elasticity in performance vs. cost.
Apache HBase is an open source, distributed, large-scale data store that provides low latency for random reads and writes. In this tutorial, learn how to create and query HBase tables with HDInsight.
Storm in HDInsight allows you to process streaming data in real time. As a managed cluster integrated into the Azure ecosystem, Storm can be configured to work with other Azure services for a complete real-time data processing and analysis solution.
Learn how to set up and run a word count topology on Storm in HDInsight. This tutorial will guide you through provisioning a Storm cluster, and then running, monitoring, and stopping a Storm topology.
Learn how to use the Microsoft HDInsight Emulator for Azure, which provides a local development environment.
Keep up to date with improvements and updates in each HDInsight release. Learn about changes you may need to make to your HDInsight configuration, jobs, and so on.
Learn how to install and use HDInsight Tools for Visual Studio to connect to HDInsight and run Hive queries.
The four samples included are intended to get you started quickly and to give you an extensible testing bed to work through concepts. Create data sets, and observe the effects of data size on jobs.
Use Script Actions to install additional software such as R and Spark or to change the configuration of applications running on an Hadoop cluster.
Customize HDInsight clusters to install additional components using custom scripts.
Data scientists and analysts can use R for big data processing on Hadoop clusters deployed in HDInsight.
Use Script Actions to install Spark on HDInsight clusters.
Follow this end-to-end scenario to learn how to develop and test a word-counting MapReduce job on HDInsight Emulator, and then deploy and run it on HDInsight.
Learn how to develop and test a Hadoop streaming MapReduce program on HDInsight Emulator, and then run it on HDInsight using a PowerShell script.
Learn how to create and build an Apache HBase application in Java using Apache Maven. Then, use the application with HDInsight (Hadoop.)
Learn how to submit MapReduce and Hive jobs using PowerShell and HDInsight .NET SDK.
Learn how to use Azure PowerShell from your workstation to submit a MapReduce program that counts word occurrences in text to an HDInsight cluster.
Use HiveQL to query data in an Apache log4j log file, and report basic statistics.
Write Pig Latin statements to analyze an Apache log4j log file, and run various queries on the data to generate output.
Both Hive and Pig allow you to create User Defined Functions (UDF) using a variety of programming languages. Learn how to use a Python UDF from Hive and Pig.
Learn how to use Azure PowerShell from a workstation to run Sqoop import and export between an HDInsight cluster and an Azure SQL database.
Learn how to run an Apache Oozie workflow to process a log4j log file to count the occurrences of each log level type. Then, export the results to an Azure SQL database table.
Learn how to use the Microsoft .NET Library for Avro to serialize objects and other data structures into streams of bytes in order to persist them to memory, a database, or a file.
Configure and run jobs on HDInsight clusters using Azure HDInsight PowerShell. Develop applications that manage HDInsight jobs with the Azure HDInsight .NET SDK.
Learn about errors that can occur when using PowerShell to manage HDInsight and the steps for recovering from them.
Learn how to develop streaming data processing applications with SCP.NET and C# on Storm in HDInsight.
Learn how to build a solution that uses a Storm cluster in HDInsight to process sensor data from Azure Event Hubs, and then displays the processed sensor data as near-real-time information on a web-based dashboard.
Learn how to analyze sensor data using Hive with HDInsight (Hadoop), and then visualize the data in Microsoft Excel. In this sample, you'll use Hive to process historical data produced by HVAC systems to see which systems can't reliably maintain a set temperature.
Learn how to use HiveQL in HDInsight to analyze website logs to get insight into the frequency of visits in a day from external websites, and a summary of website errors that the users experience.
Learn how to use a Mahout recommendation engine to get movie recommendations based on user preferences. Mahout is a machine learning library for Apache Hadoop that contains algorithms for processing data, such as filtering, classification, and clustering.
Learn how to use Apache Giraph to find the shortest path between objects using Hadoop on HDInsight. With Giraph, you can gain insights into relationships, such as between friends on social networks (also called a "social graph") or between routers on a large network like the internet.
Learn how to use Hive to analyze Twitter data to find usage frequency of a particular word.
Using geo-tagged Tweets, learn how to do real-time sentiment analysis of big data using HBase in an HDInsight (Hadoop) cluster. Then, plot statistical results on Bing Maps.
Learn how to use Hive to calculate the average flight delay among airports, and how to use Sqoop to export the results to SQL Database.
Import data from Azure HDInsight into Excel using the Microsoft Hive ODBC Driver.
A key feature of Microsoft’s Big Data solution is solid integration of Apache Hadoop with Microsoft Business Intelligence (BI) components. Learn how to use Power Query to import HDInsight data into Excel.
HDInsight clusters are enhanced to provide the reliability and availability required to manage enterprise workloads.
Learn how to use Azure Management Portal to create an HDInsight cluster, and how to open the administrative tools.
Learn how to manage HDInsight clusters using a local Azure PowerShell console.
Learn how to use the Cross-Platform Command-Line Interface to manage HDInsight clusters.
Use the Apache Ambari APIs for provisioning, managing, and monitoring Hadoop clusters. Ambari has intuitive operator tools and robust APIs that hide the complexity of Hadoop.
Learn how to define workflows and coordinators, and how to trigger the Hadoop jobs based on time.
Learn how to programmatically enumerate the YARN applications that have completed on a Hadoop cluster and access the application logs.
Learn how to provision HDInsight clusters using Azure Management Portal, PowerShell, the command-line interface, and the HDInsight .NET SDK.
Create an HBase cluster in HDInsight on Azure Virtual Network. Virtual network integration allows applications to communicate with HBase directly, improving performance and security.
Learn how to upload and access data in HDInsight using Azure Storage Explorer, Azure PowerShell, the Hadoop command line, or Sqoop.
Learn how HDInsight works with data that is stored in Azure Blob storage, when to store data in HDFS, and when to store it in Blob storage.