{"id":1880,"date":"2018-12-10T00:00:00","date_gmt":"2018-12-10T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines"},"modified":"2023-05-11T15:35:26","modified_gmt":"2023-05-11T22:35:26","slug":"deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/","title":{"rendered":"Deploying Apache Airflow in Azure to build and run data pipelines"},"content":{"rendered":"<p><a href=\"https:\/\/airflow.readthedocs.io\/en\/latest\/project.html\" target=\"_blank\" rel=\"noopener\">Apache Airflow<\/a> is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities. Since its addition to Apache foundation in 2015, Airflow has seen great adoption by the community for designing and orchestrating <a href=\"https:\/\/blog.socialcops.com\/technology\/data-science\/apache-airflow-disease-outbreaks-india\/\" target=\"_blank\" rel=\"noopener\">ETL pipelines<\/a> and <a href=\"https:\/\/engineering.siftscience.com\/sift-trains-thousands-models-using-apache-airflow\/\" target=\"_blank\" rel=\"noopener\">ML workflows<\/a>. In Airflow, a workflow is defined as a Directed Acyclic Graph (DAG), ensuring that the defined tasks are executed one after another managing the dependencies between tasks.<\/p>\n<p>A simplified version of the Airflow architecture is shown below. It consists of a web server that provides UI, a relational metadata store that can be a MySQL\/PostgreSQL database, persistent volume that stores the DAG files, a scheduler, and worker process.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" alt=\"Airflow architecture diagram\" height=\"380\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp\" title=\"Airflow architecture \" width=\"640\"><\/p>\n<p>The above architecture can be implemented to run in four execution modes, including:<\/p>\n<ul>\n<li><strong>Sequential Executor <\/strong>\u2013 This mode is useful for dev\/test or demo purpose. It serializes the operations and allows only a single task to be executed at a time.<\/li>\n<li><strong>Local Executor<\/strong> \u2013 This mode supports parallelization and is suitable for small to medium size workload. It doesn\u2019t support scaling out.<\/li>\n<li><strong>Celery Executor<\/strong> \u2013 This is the preferred mode for production deployments and is one of the ways to scale out the number of workers. For this to work, an additional celery backend which is a RabbitMQ or Redis broker is required for coordination.<\/li>\n<li><strong>Dask Executor<\/strong> \u2013 This mode also allows scaling out by leveraging the <a href=\"https:\/\/distributed.readthedocs.io\/en\/latest\/\" target=\"_blank\" rel=\"noopener\">Dask.distributed<\/a> library, allowing users to run the task in a distributed cluster.<\/li>\n<\/ul>\n<p>The above architecture can be implemented in Azure VMs or by using the managed services in Azure as shown below. For production deployments, we recommend leveraging managed services with built-in high availability and elastic scaling capabilities.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" alt=\"Image 2\" height=\"389\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/520bae1c-a084-4e9e-8fc5-1ce6041538f8.webp\" title=\"Image 2\" width=\"514\"><\/p>\n<p><a href=\"https:\/\/hub.docker.com\/r\/puckel\/docker-airflow\/\" target=\"_blank\" rel=\"noopener\">Puckel&#8217;s Airflow docker image<\/a> contains the latest build of Apache Airflow with automated build and release to the public DockerHub registry. Azure App Service for Linux is integrated with public DockerHub registry and allows you to run the Airflow <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/app-service\/containers\/app-service-linux-ci-cd\" target=\"_blank\" rel=\"noopener\">web app on Linux containers with continuous deployment<\/a>. Azure App Service also allow <a href=\"https:\/\/review.docs.microsoft.com\/en-us\/azure\/app-service\/containers\/app-service-linux-faq#multi-container-with-docker-compose-and-kubernetes\" target=\"_blank\" rel=\"noopener\">multi-container deployments with docker compose<\/a> and Kubernetes useful for celery execution mode.<\/p>\n<p>We have developed the <a href=\"https:\/\/github.com\/savjani\/azure-quickstart-templates\/tree\/master\/101-webapp-linux-airflow-postgresql\" target=\"_blank\" rel=\"noopener\">Azure QuickStart template<\/a>, which allows you to quickly deploy and create an Airflow instance in Azure by using Azure App Service and an instance of Azure Database for PostgreSQL as a metadata store.<\/p>\n<p><a href=\"https:\/\/portal.azure.com\/#create\/Microsoft.Template\/uri\/https%3A%2F%2Fraw.githubusercontent.com%2Fsavjani%2Fazure-quickstart-templates%2Fmaster%2F101-webapp-linux-airflow-postgresql%2Fazuredeploy.json\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" alt=\"Deploy to Azure icon\" height=\"34\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ee647314-4cec-4f96-be74-f6df77631bd8.webp\" title=\"Deploy to Azure icon\" width=\"161\"><\/a><\/p>\n<p>The QuickStart template automatically downloads and deploys the latest Docker container image from <a href=\"https:\/\/hub.docker.com\/r\/puckel\/docker-airflow\/\" target=\"_blank\" rel=\"noopener\">puckel\/docker-airflow<\/a> and initializes the database in Azure Database for PostgreSQL server as shown in the following graphic:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" alt=\"2018-11-25_18-28-28\" height=\"771\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/d2d5a783-4116-46b0-9c94-1a247bf4df8e.webp\" title=\"2018-11-25_18-28-28\" width=\"1676\"><\/p>\n<p>The <a href=\"https:\/\/airflow.readthedocs.io\/en\/latest\/howto\/set-config.html#setting-configuration-options\" target=\"_blank\" rel=\"noopener\">environment variables<\/a> for the Airflow docker image can be set using application settings in Azure App Service as shown in the following graphic:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" alt=\"2018-11-25_18-41-54\" height=\"726\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/552c0eaa-b035-44af-9ac5-7755c59464cc.webp\" title=\"2018-11-25_18-41-54\" width=\"1662\"><\/p>\n<p>The environment variables used in the deployment are:<\/p>\n<ul>\n<li><strong>AIRFLOW__CORE__SQL_ALCHEMY_CONN<\/strong> \u2013 Sets the connection string for web app to connect to Azure Database for PostgreSQL.<\/li>\n<li><strong>AIRFLOW__CORE__LOAD_EXAMPLES<\/strong> \u2013 Set to true to load DAG examples during deployment.<\/li>\n<\/ul>\n<p>The application setting <strong>WEBSITES_ENABLE_APP_SERVICE_STORAGE<\/strong> is set to true which can be used as a persistent storage for DAG files accessible to scheduler and worker container images.<\/p>\n<p>After it is deployed, you can browse the web server UI on port 8080 to see and monitor the DAG examples as shown in the following graphic:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" alt=\"2018-11-25_18-48-04\" height=\"839\" src=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/12ee025b-6f27-4b7b-8d18-81f7505817c3.webp\" title=\"2018-11-25_18-48-04\" width=\"1895\"><\/p>\n<p>You are now ready to orchestrate and design data pipelines for ETL and machine learning workflows by leveraging the Airflow operators.\u00a0\u00a0<\/p>\n<h2>Next steps<\/h2>\n<ul>\n<li>The above template provides a good quick start solution for anyone looking to quickly run and deploy Apache Airflow on Azure in sequential executor mode for testing and proof of concept study. However, the template is not designed for enterprise production deployments and requires expert knowledge of Azure app services and container deployments to run in Celery Executor mode. This is where we partnered with Bitnami to help simplify production grade deployments of Airflow on Azure for customers. For more details on Bitnami\u2019s multi-tier Airflow template, we request you to visit our latest <a href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/bitnami-apache-airflow-multi-tier-now-available-in-azure-marketplace\" target=\"_blank\" rel=\"noopener\">blog post<\/a>.<\/li>\n<li>If you are looking for an exciting challenge, you can deploy the <a href=\"https:\/\/github.com\/mumoshu\/kube-airflow\" target=\"_blank\" rel=\"noopener\">kube-airflow image<\/a> with celery executor with Azure Kubernetes Services using helm charts, Azure Database for PostgreSQL, and RabbitMQ. Let us know if you have developed it and we would be happy to link it to this blog.<\/li>\n<\/ul>\n<h2>Acknowledgements<\/h2>\n<p>Special thanks to Mark Bolz and Jim Toland for their contributions to the postings.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ms_queue_id":[],"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","_alt_title":"","footnotes":"","msx_community_cta_settings":[]},"categories":[1473],"tags":[],"audience":[3057,3055,3056],"content-type":[1511],"product":[1565],"tech-community":[],"topic":[],"coauthors":[528],"class_list":["post-1880","post","type-post","status-publish","format-standard","hentry","category-databases","audience-data-professionals","audience-developers","audience-it-implementors","content-type-best-practices","product-azure-database-for-postgresql","review-flag-1680286581-295","review-flag-machi-1680286585-314","review-flag-ml-1680286585-776"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Deploying Apache Airflow in Azure to build and run data pipelines | Microsoft Azure Blog<\/title>\n<meta name=\"description\" content=\"Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Deploying Apache Airflow in Azure to build and run data pipelines | Microsoft Azure Blog\" \/>\n<meta property=\"og:description\" content=\"Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Azure Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/microsoftazure\" \/>\n<meta property=\"article:published_time\" content=\"2018-12-10T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-11T22:35:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp\" \/>\n<meta name=\"author\" content=\"Parikshit Savjani\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@azure\" \/>\n<meta name=\"twitter:site\" content=\"@azure\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Parikshit Savjani\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\"},\"author\":[{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/parikshit-savjani\/\",\"@type\":\"Person\",\"@name\":\"Parikshit Savjani\"}],\"headline\":\"Deploying Apache Airflow in Azure to build and run data pipelines\",\"datePublished\":\"2018-12-10T00:00:00+00:00\",\"dateModified\":\"2023-05-11T22:35:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\"},\"wordCount\":750,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp\",\"articleSection\":[\"Databases\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\",\"name\":\"Deploying Apache Airflow in Azure to build and run data pipelines | Microsoft Azure Blog\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp\",\"datePublished\":\"2018-12-10T00:00:00+00:00\",\"dateModified\":\"2023-05-11T22:35:26+00:00\",\"description\":\"Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.\",\"breadcrumb\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog home\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Databases\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/databases\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Deploying Apache Airflow in Azure to build and run data pipelines\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"name\":\"Microsoft Azure Blog\",\"description\":\"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.\",\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\",\"name\":\"Microsoft Azure Blog\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"width\":512,\"height\":512,\"caption\":\"Microsoft Azure Blog\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/microsoftazure\",\"https:\/\/x.com\/azure\",\"https:\/\/www.instagram.com\/microsoftdeveloper\/\",\"https:\/\/www.linkedin.com\/company\/16188386\",\"https:\/\/www.youtube.com\/user\/windowsazure\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117\",\"name\":\"shakir\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"caption\":\"shakir\"},\"sameAs\":[\"https:\/\/azure.microsoft.com\"],\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Deploying Apache Airflow in Azure to build and run data pipelines | Microsoft Azure Blog","description":"Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/","og_locale":"en_US","og_type":"article","og_title":"Deploying Apache Airflow in Azure to build and run data pipelines | Microsoft Azure Blog","og_description":"Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.","og_url":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/","og_site_name":"Microsoft Azure Blog","article_publisher":"https:\/\/www.facebook.com\/microsoftazure","article_published_time":"2018-12-10T00:00:00+00:00","article_modified_time":"2023-05-11T22:35:26+00:00","og_image":[{"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp","type":"","width":"","height":""}],"author":"Parikshit Savjani","twitter_card":"summary_large_image","twitter_creator":"@azure","twitter_site":"@azure","twitter_misc":{"Written by":"Parikshit Savjani","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#article","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/"},"author":[{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/parikshit-savjani\/","@type":"Person","@name":"Parikshit Savjani"}],"headline":"Deploying Apache Airflow in Azure to build and run data pipelines","datePublished":"2018-12-10T00:00:00+00:00","dateModified":"2023-05-11T22:35:26+00:00","mainEntityOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/"},"wordCount":750,"commentCount":0,"publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp","articleSection":["Databases"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/","name":"Deploying Apache Airflow in Azure to build and run data pipelines | Microsoft Azure Blog","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp","datePublished":"2018-12-10T00:00:00+00:00","dateModified":"2023-05-11T22:35:26+00:00","description":"Apache Airflow is an open source platform used to author, schedule, and monitor workflows. Airflow overcomes some of the limitations of the cron utility by providing an extensible framework that includes operators, programmable interface to author jobs, scalable distributed architecture, and rich tracking and monitoring capabilities.","breadcrumb":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#primaryimage","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2018\/12\/ba8a473d-3f75-4735-8cf2-7f2cf54e7177.webp"},{"@type":"BreadcrumbList","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/deploying-apache-airflow-in-azure-to-build-and-run-data-pipelines\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog home","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/"},{"@type":"ListItem","position":2,"name":"Databases","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/databases\/"},{"@type":"ListItem","position":3,"name":"Deploying Apache Airflow in Azure to build and run data pipelines"}]},{"@type":"WebSite","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","name":"Microsoft Azure Blog","description":"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.","publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization","name":"Microsoft Azure Blog","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","width":512,"height":512,"caption":"Microsoft Azure Blog"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/microsoftazure","https:\/\/x.com\/azure","https:\/\/www.instagram.com\/microsoftdeveloper\/","https:\/\/www.linkedin.com\/company\/16188386","https:\/\/www.youtube.com\/user\/windowsazure"]},{"@type":"Person","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117","name":"shakir","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4","url":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","caption":"shakir"},"sameAs":["https:\/\/azure.microsoft.com"],"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/"}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Azure Blog","distributor_original_site_url":"https:\/\/azure.microsoft.com\/en-us\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/1880","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/comments?post=1880"}],"version-history":[{"count":0,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/1880\/revisions"}],"wp:attachment":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/media?parent=1880"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/categories?post=1880"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tags?post=1880"},{"taxonomy":"audience","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/audience?post=1880"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/content-type?post=1880"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/product?post=1880"},{"taxonomy":"tech-community","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tech-community?post=1880"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/topic?post=1880"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/coauthors?post=1880"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}