Gaining insights rapidly from data is critical to competitiveness in today’s business world. Azure SQL Data Warehouse (SQL DW), Microsoft’s fully managed analytics platform, leverages Massively Parallel Processing (MPP) to run complex interactive SQL queries at every level of scale.
An enterprise data warehouse doesn’t exist in isolation but is part of a complete analytics solution that includes the ‘pipelines’ that extract, transform, and load data, often from many disparate sources, as well as staging and reporting layers. Tasks such as continuous integration and continuous delivery require extensive manual coding and doesn't offer the flexibility needed for the rapidly changing business needs. This is where data warehouse automation tools provide value for customers.
Data warehouse automation (DWA) tools are meta-data driven, code generation tools that streamline developing and managing a data warehouse solution. DWA tools provide more than just ETL automation, they automate the complete life cycle of a data warehouse solution, from analysis, design, and implementation to documentation, monitoring and maintenance.
Benefits of data warehouse automation
Data warehouse automation (DWA) tools bring the benefits of meta-data driven automation, and code generation to streamline developing and managing a data warehouse solution. DWA tools provide more than just ETL automation, they automate the complete life cycle of a data warehouse solution, from analysis, design, and implementation to documentation, monitoring and maintenance.
Acceleration: By automating repeatable, tedious tasks such as source discovery, fact and dimension mapping, dimension type assignment, change-impact analysis and version control, DWA tools accelerate time to market for data warehouse projects.
“WhereScape automates the design, development and operation of agile data warehouse projects by increasing developer productivity by 10x. It leverages the enormous processing power of Azure SQL DW’s MPP architecture together with the ease of use and support that comes with using T-SQL as the processing and query language.”
– Douglas Barrett, Senior Solutions Architect, WhereScape
Consistency: By leveraging data warehouse design patterns and platform specific best practices, these tools deliver consistent, high quality code. Automatically-generated documentation improves understanding and helps with troubleshooting.
“It’s exciting to see BimlFlex customers benefiting from our Azure SQL DW automation experience. We provide optimal implementations that employ patterns approved by industry experts and Microsoft product teams - and can be selected or changed through simple modifications to our metadata store.”
– Scott Currie, CEO, Varigence
Simplicity: Graphical user interface (GUI) with point-and-click design capability offered by most DWA tools, simplify the user experience for source exploration, modelling fact and dimension tables, designing ETL pipelines and reporting.
Flexibility: By shortening iteration cycles, DWA tools help IT and business teams focus on business requirements and reduce the time-to-insight, increasing the business value of data warehousing.
“AnalytiX DS Epicenter’s metadata driven automation platform allows code to be automatically generated to truly accelerate time to value for customers while automatically documenting the data integration processes in Epicenter’s enterprise data mapping platform”
– Michael Boggs, CTO and Founder of AnalytiX DS
SQL DW and data warehouse automation partners
The following data warehouse automation partners have certified their tools with SQL DW:
“We help data driven companies be more successful with their data management. Azure SQL DW along with our powerful automation approach for designing, generating and deploying analytical solutions enables our customer to get maximum value out of their data. biGENiUS delivers optimized generator leveraging massively parallel processing capabilities of Azure SQL DW.”
– Gregor Zeiler, Senior Solution Manager, biGENiUS
The following partners have support for SQL DW coming soon:
“We are excited to be providing support for Azure SQL DW in our efforts to align our road-map with our customer’s choice of cloud data warehouse. Dimodelo Architect will speed up the transition, migrate existing data warehouse data, and automate development processes to realize the value from their cloud investment faster.”
– Adam Gilmore, Chief Product Officer, Dimodelo Solutions
“Discovery Hub by TimeXtender is a modern software platform that allows customers to build, deploy and manage a complete enterprise-grade, analytical architecture. We are pleased to be adding support for Azure DW, to complement our already broad Azure Data Services offering, in our Fall 2018 Discovery Hub release.”
– Heine Krog Iverson, CEO TimeXtender
Read what customers have to say about data warehouse automation tools:
“We started our cloud journey last year with SQL Data Warehouse to leverage its MPP architecture and the ease of scaling. It is fast becoming the backbone of all our corporate insights. To accelerate the development and to provide consistency across loading packages, we decided to use Varigence's BimlFlex as it ticked all boxes for our acceptance criteria for loading patterns, technologies and prices. We have currently been using BimlFlex to load our production Azure SQL DW for over six months.”
– Stephen Townsend, Business Intelligence Architect, an energy company
“With data warehouse automation, I define structural patterns that I consistently reuse - almost like a library I can import. It allows my team at Ashley to spend less time focusing on building out specific ETL and DW processes in the code and more time collaborating with end user on their business needs.”
– Phil Haddad, Data Architect, Ashley Furniture
“At Smurfit Kappa, we have been using biGENiUS to improve and maintain our reporting system that we have developed significantly over the last ten years. Reduction in development time, simplified loading experience, ease of disparate source addition and traceability are some of the key benefits that we gained from using a data warehouse automation tool.”
– Christian Foucher, Manager BI and Shared Applications, Smurfit Kappa
“With AnalytiX DS Enterprise Mapping and Automation platform we were able to reduce overall project duration by close to 25 percent and cost by close to 35 percent. Our deliverable’s have also improved considerably due to fact all development resources now have a common interface to a centralized metadata repository which provides us with complete process, flow and version control capability.”
– Tony Jurek, Practice Principal, DXC Technology