{"id":4196,"date":"2017-01-25T00:00:00","date_gmt":"2017-01-25T00:00:00","guid":{"rendered":""},"modified":"2025-06-11T06:33:02","modified_gmt":"2025-06-11T13:33:02","slug":"modern-data-warehousing-with-continuous-integration","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/","title":{"rendered":"Modern Data Warehousing with Continuous Integration"},"content":{"rendered":"\n

This post was co-authored by Mohit Chand, Principal SWE Lead, Microsoft IT. <\/em><\/p>\n\n\n\n

We are going a step beyond the traditional methods of developing a data warehouse by adopting CI practices, which are more prevalent for API (.NET) based applications. It has been long pending for data warehouse teams to catch up on the modern software engineering practices. With the emergence of Visual Studio Online (VSTS) & SQL Server Data Tools (SSDT), spinning environments on the fly, deploying code across environments with maximum automation has become easy. We adopted these modern practices to boost engineering productivity in our Business Insights (DW) project. With the help of SSDT and VSTS, we were able to align DW deployment perfectly with the Agile releases (2 weeks sprint). In this blog I will elaborate a detailed approach on how to implement CI for your Data Warehouse. I will explain the life cycle of a business user story starting from code branching, pull-request-triggered-build, Azure resources and environment provisioning, schema deployment, seed data generation, daily-integration releases with automated tests, and approval based workflows to promote new code to higher environments.<\/p>\n\n\n\n

DevOps<\/h2>\n\n\n\n

Why DevOps? In traditional development and operations model there is always a possibility of confusion and debate when the software doesn\u2019t function as expected. Dev would claim the software working just fine in their respective environment and defend that as an Ops problem. Ops would indicate that Devs didn\u2019t provide a production ready software, and it\u2019s a Dev problem. How do we solve this? Wouldn\u2019t it be a good idea for a single team takes care of development, testing, and operations?<\/p>\n\n\n\n

We work closely with business and other stake holders to efficiently deliver better and faster results to customers. DevOps has enabled us to deliver faster with a better a connection with customers while simultaneously reducing our technical debt and risks.<\/p>\n\n\n\n

\"DevOps\"<\/figure>\n\n\n\n

DW Architecture<\/h2>\n\n\n\n

This Data Warehouse uses Azure technologies. Data arrives to the landing zone or staging area from different sources through Azure Data Factory. We use Azure Data Factory (ADF) jobs to massage and transform data into the warehouse. Once ready, the data is available to customers in the form of dimension and fact tables.<\/p>\n\n\n\n

\"DWArchitecture\"<\/figure>\n\n\n\n

Tools\/Technologies<\/h3>\n\n\n\n

This Modern Data Warehouse primarily uses Microsoft technologies to deliver the solution to customers –<\/p>\n\n\n\n

    \n
  • SQL Azure (PaaS)<\/li>\n\n\n\n
  • Azure Data Factory<\/li>\n\n\n\n
  • Azure Blob Storage<\/li>\n\n\n\n
  • SQL MDS 2016<\/li>\n\n\n\n
  • Visual Studio Team Services (VSTS)\n
      \n
    • \n
        \n
      • Agile and Kanban board<\/li>\n\n\n\n
      • Code branching (Git)<\/li>\n\n\n\n
      • Gated check-ins<\/li>\n\n\n\n
      • Automated Tests<\/li>\n\n\n\n
      • Build<\/li>\n\n\n\n
      • Release<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n

        Plan<\/h2>\n\n\n\n

        In agile scrum, user story is the unit of implementation. Engineers pick up and deliver the user stories in any given sprint.<\/p>\n\n\n\n

        \"Story\"<\/figure>\n\n\n\n

        Code<\/h2>\n\n\n\n

        Branching Strategy<\/h3>\n\n\n\n

        With Agile code branching plays a critical role. There are various ways to do it including sprint branching, feature branching, story\/bug branching, etc. In our case we adopted user story level branching. A contributor creates a branch from the \u201cdevelop\u201d branch against each story he\/she picks up. It is the contributor\u2019s responsibility to maintain this isolated branch and merge it by creating a \u201cPull Request\u201d with the develop branch once the story is complete or ready for code review. A contributor is not allowed to merge his\/her code with the main stream branch directly. It requires a minimum two code reviews to approve the code.<\/p>\n\n\n\n

        Story based branching enables developers to merge the code frequently<\/strong> with the main stream and avoid working a long time on the same branch. This significantly reduces the code integration issues. Another benefit is that developers can work more efficiently by having other developer\u2019s code available more frequently. Code dependency wait time gets reduced, hence less blockers for developers.<\/p>\n\n\n\n

        Using VSTS, contributor creates a new branch:<\/p>\n\n\n\n

        \"Branch-1\"<\/figure>\n\n\n\n

        Nomenclature followed is _<\/p>\n\n\n\n

        \"Branch-2\"<\/figure>\n\n\n\n

        After creating the branch, the contributor publishes it to make it visible to everyone else on the team.<\/p>\n\n\n\n

        \"Branch-3\"<\/figure>\n\n\n\n

        Once the branching is set the contributor is all set to start with the story and starts writing code to implement the functionality.<\/p>\n\n\n\n

        \"Checkin-1\"<\/figure>\n\n\n\n

        Code Review and Code Merge<\/h3>\n\n\n\n

        Once the code is complete, the developer checks-in the code and creates a pull request using VSTS portal. To ensure a higher level of code quality, it\u2019s imperative to have a gated check-in process in place. Each developer has to ensure the build is not broken when they check-in the code. The code needs to be reviewed and approved by at least two peers before it gets merged with the main stream. Without two code review approvals, it\u2019s not possible to merge the code with the main stream code.<\/p>\n\n\n\n

        \"PullRequest-1\"<\/figure>\n\n\n\n

        A pull request is created by the developer and submitted with appropriate comments and work items.<\/p>\n\n\n\n

        \"PullRequest-2\"<\/figure>\n\n\n\n

        Build<\/h3>\n\n\n\n

        As soon as the \u201cPull Request\u201d is created by the contributor, the CI build automatically gets fired.<\/p>\n\n\n\n

        \"Build-1\"<\/figure>\n\n\n\n
        \"Build-2\"<\/figure>\n\n\n\n

        An email notification is received by all the reviewers, which updates them about the new \u201cPull Request\u201d created by a contributor. The reviewers are now good to starts the code reviews.<\/p>\n\n\n\n

        \"CodeReview-1\"<\/figure>\n\n\n\n

        <\/p>\n\n\n\n

        \"CodeReview-2\"<\/figure>\n\n\n\n

        Depending upon the quality of the code, the reviewer \u201cApproves\u201d, raise questions, or \u201cReject\u201d the code.<\/p>\n\n\n\n

        \"CodeReview-3\"<\/figure>\n\n\n\n

        Once all reviewers are done with code reviews, the lead developer merges the code with the main stream.<\/p>\n\n\n\n

        \"CodeReview-4\"<\/figure>\n\n\n\n

        Test (Automation)<\/h2>\n\n\n\n

        Ensuring high code quality across various environments could be challenging in a DW project because data might vary from environment to environment. We ensure that every new piece of code we write has automated test cases before creating a pull request. This not only prevented bug leaks to production, but it also ensured a higher quality of deliverables.<\/p>\n\n\n\n

        The diagram below depicts the overall test case execution results. As part of the deployment, we execute all of our test cases to ensure the integrity and quality of the end product.<\/p>\n\n\n\n

        \"TestCases-1\"<\/figure>\n\n\n\n

        Release & Deploy<\/h2>\n\n\n\n

        Once the code is successfully merged with main stream code, a new build fires automatically. The Integration environment gets deployed once a day with the latest code.<\/p>\n\n\n\n

        \"Release-1\"<\/figure>\n\n\n\n

        The diagram below depicts three environments we manage for the Data Warehouse. We have \u201cIntegration\u201d, \u201cEnd User\u201d, and \u201cProduction\u201d environments. The integration environment is a continuous integration and deployment environment, which is provisioned and de-provisioned dynamically and managed as \u201cInfra as a Code<\/strong>\u201d. It is a scheduled process which run the following steps in sequence to \u201cintegrate\u201d the check-ins happening daily.<\/p>\n\n\n\n

          \n
        • Build the new bits by getting the latest from the develop branch (includes integrated code scanning)<\/li>\n\n\n\n
        • Create a new Azure Resource Group and procure the SQL Instance<\/li>\n\n\n\n
        • Copy the \u201cSeed Data\u201d to the newly created SQL instance<\/li>\n\n\n\n
        • Execute any schema renaming<\/li>\n\n\n\n
        • Deploy the DACPAC to deploy new schema changes<\/li>\n\n\n\n
        • Scale-up the databases to execute the steps faster<\/li>\n\n\n\n
        • Copy code bits to build server<\/li>\n\n\n\n
        • Deploy additional SQL entities<\/li>\n\n\n\n
        • Run data sync jobs<\/li>\n\n\n\n
        • Execute test assemblies<\/li>\n\n\n\n
        • Deploy Azure Data Factories<\/li>\n\n\n\n
        • Decommission environment<\/li>\n<\/ul>\n\n\n\n
          \"Release-2\"<\/figure>\n\n\n\n

          Seed data to enable automated testing<\/h3>\n\n\n\n

          Automated testing in a DW depends a lot on the availability of accurate data. Without data there will be numerous scenarios which cannot be tested. To solve this problem we use production copy of data as \u201cseed data\u201d during deployment. The diagram below depicts how we populate the seed data in our Daily Integration Environment (DIT). In cases where the data is pretty huge, a miniature DB, which contains subset of production data, can be used instead of copying the entire replica.<\/p>\n\n\n\n

          \"SeedData\"<\/figure>\n\n\n\n

          <\/p>\n\n\n\n

          Step-1: Represents the production data with multiple schemas which we use to segregate data in our DW environments (e.g. staging, transformed data, etc.)<\/p>\n\n\n\n

          Step-2: Represents that data gets copied to Azure Geo Replica (Disaster Recovery copy).<\/p>\n\n\n\n

          Step-3: During release deployment we copy the geo replica to the newly procured DIT SQL server instance.<\/p>\n\n\n\n

          Step-4: Represents the availability of production equivalent copy of data. In addition, the DACPACK deployment happens to add the newly added schema and later our test automation suites runs to test the quality of our end product.<\/p>\n\n\n\n

          Deploying release to higher environments<\/h4>\n\n\n\n

          Promoting the release from one environment to another is setup through an approval workflow and it is not allowed to deploy directly. In this scenario it is not possible to deploy directly to Production without the approval from pre-assigned stakeholders.<\/p>\n\n\n\n

          \"WorkFlow-1\"<\/figure>\n\n\n\n

          <\/p>\n\n\n\n

          The approval workflow depicts the environment promotion. Once the required approvers approve the workflow the release gets promoted to next environment automatically.<\/p>\n\n\n\n

          \"Approval-1\"<\/figure>\n\n\n\n

          Monitor<\/h2>\n\n\n\n

          As DevOps the same team monitors the pre-prod and prod environments for any failures. We adopted the DRI (directly responsible individual) model and the DRI person proactively monitors and checks the system\u2019s health and all the notifications. Issues, if any, are fixed on priority to ensure continued availability of application. We use out of the box ADF monitoring and notification along with a couple of custom monitoring tools. We also have multiple data quality checks implemented as automated reports that run daily in the production environment and report out data anomalies that can either be fixed as a bug in our processes or be traced back to the source systems quickly for getting fixed there.<\/p>\n\n\n\n

          It\u2019s true that setting up CI for a data warehouse isn\u2019t that simple, however, it\u2019s worth every penny. We did face challenges of test case failures when we add new code during the sprints, however, the team has learned from those instances and now we ensure to update the existing test cases when new code is being added. We are continuously adding functional, build, and environment verification test cases to constantly increase the quality of the product. CI has enabled us to be truly agile and be super confident in our end product. We are able to prevent common bug leaks to production by having an automated test suite. We were able to eliminate the need of the test environment and eyeing to deploy directly in production in coming quarters. We strongly believe it\u2019s possible!<\/p>\n","protected":false},"excerpt":{"rendered":"

          Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ms_queue_id":[],"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","_alt_title":"","footnotes":"","msx_community_cta_settings":[]},"categories":[1474,1473],"tags":[],"audience":[3054,3057,3055,3053,3056],"content-type":[1511],"product":[1821],"tech-community":[],"topic":[],"coauthors":[1169],"class_list":["post-4196","post","type-post","status-publish","format-standard","hentry","category-analytics","category-databases","audience-business-decision-makers","audience-data-professionals","audience-developers","audience-it-decision-makers","audience-it-implementors","content-type-best-practices","product-azure-data-factory","review-flag-1680286581-295","review-flag-1680286581-56","review-flag-1-1680286581-825","review-flag-2-1680286581-601","review-flag-3-1680286581-173","review-flag-4-1680286581-250","review-flag-alway-1680286580-106","review-flag-integ-1680286579-214","review-flag-new-1680286579-546"],"yoast_head":"\nModern Data Warehousing with Continuous Integration | Microsoft Azure Blog<\/title>\n<meta name=\"description\" content=\"Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications. It\u2019s been long pending for data warehouse\u2026\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Modern Data Warehousing with Continuous Integration | Microsoft Azure Blog\" \/>\n<meta property=\"og:description\" content=\"Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications. It\u2019s been long pending for data warehouse\u2026\" \/>\n<meta property=\"og:url\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Azure Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/microsoftazure\" \/>\n<meta property=\"article:published_time\" content=\"2017-01-25T00:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-11T13:33:02+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp\" \/>\n<meta name=\"author\" content=\"Naval Tripathi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@azure\" \/>\n<meta name=\"twitter:site\" content=\"@azure\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Naval Tripathi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\"},\"author\":[{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/naval-tripathi\/\",\"@type\":\"Person\",\"@name\":\"Naval Tripathi\"}],\"headline\":\"Modern Data Warehousing with Continuous Integration\",\"datePublished\":\"2017-01-25T00:00:00+00:00\",\"dateModified\":\"2025-06-11T13:33:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\"},\"wordCount\":1621,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp\",\"articleSection\":[\"Analytics\",\"Databases\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\",\"name\":\"Modern Data Warehousing with Continuous Integration | Microsoft Azure Blog\",\"isPartOf\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp\",\"datePublished\":\"2017-01-25T00:00:00+00:00\",\"dateModified\":\"2025-06-11T13:33:02+00:00\",\"description\":\"Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications. It\u2019s been long pending for data warehouse\u2026\",\"breadcrumb\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog home\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Analytics\",\"item\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/analytics\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Modern Data Warehousing with Continuous Integration\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#website\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"name\":\"Microsoft Azure Blog\",\"description\":\"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.\",\"publisher\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization\",\"name\":\"Microsoft Azure Blog\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"contentUrl\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp\",\"width\":512,\"height\":512,\"caption\":\"Microsoft Azure Blog\"},\"image\":{\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/microsoftazure\",\"https:\/\/x.com\/azure\",\"https:\/\/www.instagram.com\/microsoftdeveloper\/\",\"https:\/\/www.linkedin.com\/company\/16188386\",\"https:\/\/www.youtube.com\/user\/windowsazure\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117\",\"name\":\"shakir\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g\",\"caption\":\"shakir\"},\"sameAs\":[\"https:\/\/azure.microsoft.com\"],\"url\":\"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Modern Data Warehousing with Continuous Integration | Microsoft Azure Blog","description":"Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications. It\u2019s been long pending for data warehouse\u2026","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/","og_locale":"en_US","og_type":"article","og_title":"Modern Data Warehousing with Continuous Integration | Microsoft Azure Blog","og_description":"Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications. It\u2019s been long pending for data warehouse\u2026","og_url":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/","og_site_name":"Microsoft Azure Blog","article_publisher":"https:\/\/www.facebook.com\/microsoftazure","article_published_time":"2017-01-25T00:00:00+00:00","article_modified_time":"2025-06-11T13:33:02+00:00","og_image":[{"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp","type":"","width":"","height":""}],"author":"Naval Tripathi","twitter_card":"summary_large_image","twitter_creator":"@azure","twitter_site":"@azure","twitter_misc":{"Written by":"Naval Tripathi","Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#article","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/"},"author":[{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/naval-tripathi\/","@type":"Person","@name":"Naval Tripathi"}],"headline":"Modern Data Warehousing with Continuous Integration","datePublished":"2017-01-25T00:00:00+00:00","dateModified":"2025-06-11T13:33:02+00:00","mainEntityOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/"},"wordCount":1621,"commentCount":0,"publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp","articleSection":["Analytics","Databases"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/","name":"Modern Data Warehousing with Continuous Integration | Microsoft Azure Blog","isPartOf":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage"},"thumbnailUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp","datePublished":"2017-01-25T00:00:00+00:00","dateModified":"2025-06-11T13:33:02+00:00","description":"Going a step beyond the traditional ways of developing a data warehouse by adopting CI practices which are more prevalent for API (.NET) based applications. It\u2019s been long pending for data warehouse\u2026","breadcrumb":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#primaryimage","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2017\/01\/f2c166bc-ab31-4260-8167-2d06248b8ff0.webp"},{"@type":"BreadcrumbList","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/modern-data-warehousing-with-continuous-integration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog home","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/"},{"@type":"ListItem","position":2,"name":"Analytics","item":"https:\/\/azure.microsoft.com\/en-us\/blog\/category\/analytics\/"},{"@type":"ListItem","position":3,"name":"Modern Data Warehousing with Continuous Integration"}]},{"@type":"WebSite","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#website","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","name":"Microsoft Azure Blog","description":"Get the latest Azure news, updates, and announcements from the Azure blog. From product updates to hot topics, hear from the Azure experts.","publisher":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/azure.microsoft.com\/en-us\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#organization","name":"Microsoft Azure Blog","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","contentUrl":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-content\/uploads\/2024\/06\/microsoft_logo.webp","width":512,"height":512,"caption":"Microsoft Azure Blog"},"image":{"@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/microsoftazure","https:\/\/x.com\/azure","https:\/\/www.instagram.com\/microsoftdeveloper\/","https:\/\/www.linkedin.com\/company\/16188386","https:\/\/www.youtube.com\/user\/windowsazure"]},{"@type":"Person","@id":"https:\/\/azure.microsoft.com\/en-us\/blog\/#\/schema\/person\/c702e5edd662b328b49b7e1180cab117","name":"shakir","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g7664e653ea371ce16eaf75e9fa8952c4","url":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9342c7c05bb16548741bc5cd3a3e3b7ee0c8e746844ad2cc582db5beb5514c6f?s=96&d=mm&r=g","caption":"shakir"},"sameAs":["https:\/\/azure.microsoft.com"],"url":"https:\/\/azure.microsoft.com\/en-us\/blog\/author\/shakir\/"}]}},"msxcm_display_generated_audio":false,"msxcm_animated_featured_image":null,"distributor_meta":false,"distributor_terms":false,"distributor_media":false,"distributor_original_site_name":"Microsoft Azure Blog","distributor_original_site_url":"https:\/\/azure.microsoft.com\/en-us\/blog","push-errors":false,"_links":{"self":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/4196","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/comments?post=4196"}],"version-history":[{"count":1,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/4196\/revisions"}],"predecessor-version":[{"id":41452,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/posts\/4196\/revisions\/41452"}],"wp:attachment":[{"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/media?parent=4196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/categories?post=4196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tags?post=4196"},{"taxonomy":"audience","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/audience?post=4196"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/content-type?post=4196"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/product?post=4196"},{"taxonomy":"tech-community","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/tech-community?post=4196"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/topic?post=4196"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/azure.microsoft.com\/en-us\/blog\/wp-json\/wp\/v2\/coauthors?post=4196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}