{"id":3776,"date":"2017-06-15T00:00:00","date_gmt":"2017-06-15T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks"},"modified":"2025-06-20T04:48:24","modified_gmt":"2025-06-20T11:48:24","slug":"announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/","title":{"rendered":"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago <a href=\"https:\/\/www.google.com\/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=5&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwi2ge_F-LrUAhUhS5oKHWZFAg4QFghCMAQ&amp;url=https%3A%2F%2Fchannel9.msdn.com%2FShows%2FAzure-Friday%2FIntroducing-Apache-Kafka-on-Azure-HDInsight&amp;usg=AFQjCNE9K8XAt-z0ppbTkFYk6bgbsEbduw&amp;sig2=1JR8uoivaE9PURAqCjqeSg\">we announced<\/a> a limited preview of <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/hdinsight\/apache-kafka\/\">Managed Kafka on Azure HDInsight<\/a>. The addition of Kafka on HDInsight completes the ingestion piece for scalable open source streaming on Azure. In addition to the scale and performance benefits of Apache Kafka, HDInsight Kafka customers reap the following advantages:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">The promise of a managed open source Kafka backed by a <a href=\"https:\/\/azure.microsoft.com\/en-us\/support\/legal\/sla\/hdinsight\/v1_0\/?v=17.23h\">99.9%<\/a> uptime SLA\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\">This includes installation, configuration, and management of open source components<\/li>\n\n\n\n<li class=\"wp-block-list-item\">HDInsight additionally provisions and monitors a Zookeeper quorum as part of the cluster shape.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Managed rebalance of replicas and partitions across Azure update domains and fault domains. This ensures high availability of Kafka partitions on environments with a multidimensional view of a rack. This tool is also <a href=\"https:\/\/github.com\/hdinsight\/hdinsight-kafka-tools\">open sourced here<\/a>.<\/li>\n\n\n\n<li class=\"wp-block-list-item\">Security and compliance benefits of <a href=\"https:\/\/www.microsoft.com\/en-us\/TrustCenter\/Compliance\">Azure<\/a> and <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/hdinsight\/?v=17.23h\">HDInsight<\/a> with certifications such as SOC, PCI, DSS.<\/li>\n\n\n\n<li class=\"wp-block-list-item\">An integrated experience to deploy a managed and secure streaming pipeline (Kafka, Storm or Spark streaming) within minutes via prebuilt architectures on ARM templates.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Today we are pleased to announce the Public Preview of Apache Kafka with <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/managed-disks\/\">Azure Managed Disks<\/a> on the HDInsight platform. Users can now deploy Kafka clusters with managed disks straight from Azure portal, with no signup necessary. This allows for the powerful advantage of exponentially higher scalability, alongside exponential lower cost as workloads scale. This feature is discussed in more detail below.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"customer-success-stories-toyota-connected\">Customer Success Stories: Toyota Connected<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Over the last year HDInsight has worked very closely with is Toyota Inc. to build one of the world\u2019s largest and most distributed connected car streaming platform. This platform processes millions of large events\/day in production on HDInsight Kafka to unlock insights in real-time. A platform at this scale was made possible by the secure, managed, and elastically scalable nature of HDInsight. The benefits are best explained by the Chief Product Owner of Toyota Connected below.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><img loading=\"lazy\" decoding=\"async\" title=\"image\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp\" alt=\"image\" width=\"189\" height=\"121\" align=\"left\">&#8220;Toyota manufactures millions of cars running globally, and building a connected car platform to process real-time data at Toyota scale is a monumental challenge. To process events at Toyota\u2019s scale, technologies such as Kafka need to be leveraged. Since HDInsight is the only managed platform that provides Kafka as a managed service with a 99.9% SLA, Toyota was able to leverage the scalable technology of Kafka, Storm and Spark on Azure HDInsight. Using the HDInsight platform, we were able to deploy enterprise grade streaming pipelines to process events from millions of cars every second. This is just scratching the surface &#8211; the future of global connected cars on Azure HDInsight is bright, and we are excited for what&#8217;s in store.&#8221;&nbsp;&nbsp; -Vijay Chemuturi, Chief Product Owner, Toyota Connected<\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">A high-level architecture of the connected car architecture is depicted below. As Vijay states, this is just the beginning \u2013 we are very excited to build upon this powerful streaming platform in the upcoming months.<\/p>\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/ca4cbb01-9e0e-428b-8a5f-0dca84d4bcc5.webp\" alt=\"A diagram of a company logo\" class=\"wp-image-42646 webp-format\" data-orig-src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/ca4cbb01-9e0e-428b-8a5f-0dca84d4bcc5.webp\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">More details on architecting similar IoT scenarios will follow in upcoming series of blogs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"integration-of-hdinsight-kafka-with-azure-managed-disks\">Integration of HDInsight Kafka with Azure Managed Disks<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With this public preview, HDInsight Kafka is also releasing native integration with Azure Managed Disks.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Azure Managed Disks is a new feature that abstracts the storage account specification for the customer allowing for an easier and managed route to use disks. They provide for a higher scale by abolishing the storage account IOPS limitation, along with the ability to create hundreds of VMs from a given VHD in a centralized storage account. A disk can be either Premium (SSD) or Standard (HDD), and 1 TB in size. More information on this feature is located here.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka is a high throughput, low latency messaging service that is I\/O heavy. Prior to Azure Managed Disks, HDInsight Kafka\u2019s original preview offering stored data on the largest persisted disk of the node. This meant that each node had a limitation of 1 TB. Given Kafka\u2019s I\/O heavy nature the disk would often become the bottleneck and additional nodes needed to be added for more storage. This resulted in high cost, with a gross underutilization of the CPU and memory on the cluster. With this release, we are implementing the HDInsight Kafka with Managed Disks feature, which is pictorially depicted below.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/e753a130-ce6b-47cc-b104-27cad7858d30.webp\" alt=\"Kafka-MD-Diagram\" title=\"Kafka-MD-Diagram\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">With this feature, one can have both persisted and scalable data, up to 16 TBs per node. This allows for an exponentially lower cost, higher scalability and better performance as the workloads increase. Since the cost of a disk is a fraction of the cost of a node, the below figure shows how the number of nodes and cost scales down exponentially as scaling needs increase.<\/p>\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/b55a01bf-8195-4e48-9b90-4dc5b8dfb657.webp\" alt=\"Kafka Cost Estimator\" class=\"wp-image-42647 webp-format\" data-orig-src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/b55a01bf-8195-4e48-9b90-4dc5b8dfb657.webp\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">This feature is automatically turned on, and taking advantage of this feature is simple \u2013 the user just needs to specify the number of disks to be attached to a given node. This can be done via the portal, or by specifying a single property in the ARM template, shown in the below figures. Note that the type of disk \u2013 Premium or Standard is determined by the type of VMs chosen the worker nodes. Premium disks are attached to D<strong>S<\/strong> and G<strong>S<\/strong> series VMs, whereas standard are attached to all other VM types. End to end templates, with examples on how to create these clusters are detailed in the next section. More information on this is located in our <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/hdinsight\/hdinsight-apache-kafka-get-started\">documentation<\/a>.<\/p>\n\n\n\n<div class=\"wp-block-group is-nowrap is-layout-flex wp-container-core-group-is-layout-6c531013 wp-block-group-is-layout-flex\"><figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/93bc6b29-3f05-4a4c-a7b4-fe344ee1011d-1.webp\" alt=\"A screenshot of a computer\" class=\"wp-image-42648 webp-format\" data-orig-src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/93bc6b29-3f05-4a4c-a7b4-fe344ee1011d-1.webp\"><\/figure>\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/44335bed-7ad3-44cb-89ec-afc0a4662bf6.webp\" alt=\"A screen shot of a computer program\" class=\"wp-image-42649 webp-format\" style=\"object-fit:cover\" data-orig-src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/44335bed-7ad3-44cb-89ec-afc0a4662bf6.webp\"><\/figure>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Disk Specification via Portal 1<\/em>          &#8212;&#8212;                <em>Disk Specification via ARM template 1<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"start-deploying-and-using-spark-storm-and-kafka-with-managed-disks-on-hdinsight-within-minutes\">Start deploying and using Spark, Storm, and Kafka with Managed Disks on HDInsight within minutes<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We have updated our documentation and samples to help deploy scalable open source streaming solutions on HDInsight. Each of these examples walks through creating the clusters step by step, and contain one-click deploy ARM templates to enable powerful pipelines. We have additionally updated the Spark Streaming examples to include the new examples for Structured Streaming, and creating an end to end pipeline using Twitter, Kafka, Spark Streaming..<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"wp-block-list-item\"><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/hdinsight\/hdinsight-apache-kafka-get-started\">Getting Started with Kafka for HDInsight<\/a><\/li>\n\n\n\n<li class=\"wp-block-list-item\"><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/hdinsight\/hdinsight-apache-spark-with-kafka\">Deploy HDInsight Kafka + Spark streaming<\/a><\/li>\n\n\n\n<li class=\"wp-block-list-item\"><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/hdinsight\/hdinsight-apache-storm-with-kafka\">Deploy HDInsight Kafka + Storm<\/a><\/li>\n\n\n\n<li class=\"wp-block-list-item\"><a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/hdinsight\/hdinsight-apache-kafka-connect-vpn-gateway\">Stream data from on-premise to HDInsight Kafka in the cloud<\/a><\/li>\n\n\n\n<li class=\"wp-block-list-item\"><a href=\"https:\/\/docs.microsoft.com\/azure\/hdinsight\/hdinsight-apache-kafka-spark-structured-streaming\">Stream tweets to HDInsight Kafka and process with Spark structured streaming<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">For any questions, suggestions or feedback, please do not hesitate to reach out to us via <a href=\"mailto:HDIFeedback@microsoft.com\">HDIFeedback@microsoft.com<\/a>. We are really excited to have you onboard, and would love to hear from you.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ms_queue_id":[],"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","_alt_title":"","footnotes":"","msx_community_cta_settings":[]},"categories":[1482],"tags":[48],"audience":[3054,3053],"content-type":[],"product":[1672],"tech-community":[],"topic":[],"coauthors":[97],"class_list":["post-3776","post","type-post","status-publish","format-standard","hentry","category-management-and-governance","tag-big-data","audience-business-decision-makers","audience-it-decision-makers","product-azure-managed-applications","review-flag-1680286581-295","review-flag-1680286581-364","review-flag-1-1680286581-825","review-flag-9-1680286581-259","review-flag-iot-1680286585-835","review-flag-lever-1680286579-649","review-flag-new-1680286579-546","review-flag-on-pr-1680286585-498","review-flag-publi-1680286584-566","review-flag-vm-1680286585-143"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks | Microsoft Azure Blog<\/title>\n<meta name=\"description\" content=\"HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks | Microsoft Azure Blog\" \/>\n<meta property=\"og:description\" content=\"HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Azure Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/microsoftazure\" \/>\n<meta property=\"article:published_time\" content=\"2017-06-15T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-20T11:48:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp\" \/>\n<meta name=\"author\" content=\"Microsoft Azure\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@azure\" \/>\n<meta name=\"twitter:site\" content=\"@azure\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Microsoft Azure\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\"},\"author\":[{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/microsoft-azure\/\",\"@type\":\"Person\",\"@name\":\"Microsoft Azure\"}],\"headline\":\"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks\",\"datePublished\":\"2017-06-15T00:00:00+00:00\",\"dateModified\":\"2025-06-20T11:48:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\"},\"wordCount\":1071,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp\",\"keywords\":[\"Big Data\"],\"articleSection\":[\"Management and governance\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\",\"name\":\"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks | Microsoft Azure Blog\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp\",\"datePublished\":\"2017-06-15T00:00:00+00:00\",\"dateModified\":\"2025-06-20T11:48:24+00:00\",\"description\":\"HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.\",\"breadcrumb\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog home\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Management and governance\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/management-and-governance\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"name\":\"Microsoft Azure Blog\",\"description\":\"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.\",\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\",\"name\":\"Microsoft Azure Blog\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"width\":512,\"height\":512,\"caption\":\"Microsoft Azure Blog\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/microsoftazure\",\"https:\/\/x.com\/azure\",\"https:\/\/www.instagram.com\/microsoftdeveloper\/\",\"https:\/\/www.linkedin.com\/company\/16188386\",\"https:\/\/www.youtube.com\/user\/windowsazure\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117\",\"name\":\"shakir\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"caption\":\"shakir\"},\"sameAs\":[\"https:\/\/azure.microsoft.com\"],\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks | Microsoft Azure Blog","description":"HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/","og_locale":"en_US","og_type":"article","og_title":"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks | Microsoft Azure Blog","og_description":"HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.","og_url":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/","og_site_name":"Microsoft Azure Blog","article_publisher":"https:\/\/www.facebook.com\/microsoftazure","article_published_time":"2017-06-15T00:00:00+00:00","article_modified_time":"2025-06-20T11:48:24+00:00","og_image":[{"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp","type":"","width":"","height":""}],"author":"Microsoft Azure","twitter_card":"summary_large_image","twitter_creator":"@azure","twitter_site":"@azure","twitter_misc":{"Written by":"Microsoft Azure","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#article","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/"},"author":[{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/microsoft-azure\/","@type":"Person","@name":"Microsoft Azure"}],"headline":"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks","datePublished":"2017-06-15T00:00:00+00:00","dateModified":"2025-06-20T11:48:24+00:00","mainEntityOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/"},"wordCount":1071,"commentCount":0,"publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp","keywords":["Big Data"],"articleSection":["Management and governance"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/","name":"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks | Microsoft Azure Blog","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp","datePublished":"2017-06-15T00:00:00+00:00","dateModified":"2025-06-20T11:48:24+00:00","description":"HDInsight set a firm goal of helping enterprises build secure, robust, scalable open source streaming pipelines on Azure. To meet this goal, a few months ago we announced a limited preview of Managed Kafka on Azure HDInsight.","breadcrumb":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#primaryimage","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/06\/2a321347-6588-4d17-aa18-cc96ac94e872.webp"},{"@type":"BreadcrumbList","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/announcing-public-preview-of-apache-kafka-on-hdinsight-with-azure-managed-disks\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog home","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/"},{"@type":"ListItem","position":2,"name":"Management and governance","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/management-and-governance\/"},{"@type":"ListItem","position":3,"name":"Announcing public preview of Apache Kafka on HDInsight with Azure Managed disks"}]},{"@type":"WebSite","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","name":"Microsoft Azure Blog","description":"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.","publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization","name":"Microsoft Azure Blog","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","width":512,"height":512,"caption":"Microsoft Azure Blog"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/microsoftazure","https:\/\/x.com\/azure","https:\/\/www.instagram.com\/microsoftdeveloper\/","https:\/\/www.linkedin.com\/company\/16188386","https:\/\/www.youtube.com\/user\/windowsazure"]},{"@type":"Person","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117","name":"shakir","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4","url":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","caption":"shakir"},"sameAs":["https:\/\/azure.microsoft.com"],"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/"}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Azure Blog","distributor_original_site_url":"https:\/\/azure.microsoft.com\/en-us\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/3776","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/comments?post=3776"}],"version-history":[{"count":3,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/3776\/revisions"}],"predecessor-version":[{"id":42654,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/3776\/revisions\/42654"}],"wp:attachment":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/media?parent=3776"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/categories?post=3776"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tags?post=3776"},{"taxonomy":"audience","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/audience?post=3776"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/content-type?post=3776"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/product?post=3776"},{"taxonomy":"tech-community","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tech-community?post=3776"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/topic?post=3776"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/coauthors?post=3776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}