Data Factory
A fully managed, serverless data integration solution for ingesting, preparing and transforming all of your data at scale.
Hybrid data integration simplified
Integrate all of your data with Azure Data Factory – a fully managed, serverless data integration service. Visually integrate data sources with more than 90 built-in, maintenance-free connectors at no added cost. Easily construct ETL and ELT processes code-free in an intuitive environment or write your own code. Then deliver integrated data to Azure Synapse Analytics to unlock business insights.
Easy to use
Rehost SQL Server Integration Services (SSIS) in a few clicks and build ETL and ELT pipelines code-free, with built-in Git and CI/CD support.
Cost-effective
Enjoy a pay-as-you-go, fully managed serverless cloud service that scales on demand.
Powerful
Ingest all of your on-premises, and software as a service (SaaS) data with more than 90 built-in connectors. Orchestrate and monitor at scale.
Intelligent
Use autonomous ETL to unlock operational efficiencies and enable citizen integrators.
Accelerate data transformation with code-free data flows
Data Factory provides a data integration and transformation layer that works across your digital transformation initiatives.
- Enable citizen integrators and data engineers to drive business and IT led Analytics/BI.
- Prepare data, construct ETL and ELT processes, and orchestrate and monitor pipelines code-free. The managed Apache Spark™ service takes care of code generation and maintenance.
- Transform faster with intelligent intent-driven mapping that automates copy activities.
Rehost and extend SSIS in a few clicks
Azure Data Factory can help organisations looking to modernise SSIS.
- Realise up to 88 per cent cost savings with the Azure Hybrid Benefit.
- Enjoy the only fully compatible service that makes it easy to move all of your SSIS packages to the cloud.
- Migration is easy with the deployment wizard and ample how-to documentation.
- Realise your vision for hybrid big data and data warehousing initiatives by combining with Data Factory cloud data pipelines.
Ingest all of your data with built-in connectors
Ingesting data from diverse and multiple sources can be expensive, time-consuming and require multiple solutions. Azure Data Factory offers a single, pay-as-you-go service. You can:
- Choose from more than 90 built-in connectors to acquire data from Big Data sources such as Amazon Redshift, Google BigQuery, HDFS; enterprise data warehouses such as Oracle Exadata, Teradata; SaaS apps such as Salesforce, Marketo and ServiceNow; and all Azure data services.
- Use the full capacity of underlying network bandwidth, up to 5 GB/s throughput.
Get the most out of Azure Synapse Analytics
Ingest data from on-premises, hybrid and multicloud sources and transform it with powerful data flows in Azure Synapse Analytics, powered by Data Factory.
- Integrate and transform data in the familiar Data Factory experience within Azure Synapse Pipelines
- Transform and analyse data code-free with Data flows within the Azure Synapse studio.
- Integrate all of your data with more than 90 built-in connectors.
Ignite your app experiences with the right data
Data Factory can help independent software vendors (ISVs) enrich their SaaS apps with integrated hybrid data in order to deliver data-driven user experiences. Pre-built connectors and integration at scale enable you to focus on your users while Data Factory takes care of the rest.
Explore how Data Factory can help youOrchestrate, monitor and manage pipeline performance
Maintaining pipelines with a rapidly changing data landscape can quickly become time-consuming, involving manual interventions. In Azure Data Factory, you can not only monitor all of your activity runs visually, you can also improve operational productivity by setting up alerts proactively to monitor your pipelines. These alerts can then appear within Azure alert groups, ensuring that you’re notified in time to prevent downstream or upstream problems before they happen.
Learn moreHybrid data integration, simplified
Hybrid data integration, simplified
In today’s data-driven world, big data processing is a critical task for every organisation. To unlock transformational insights, data engineers need services that are built to simplify ETL as well as handle the complexities and scale challenges of big data integration.
With Azure Data Factory, it’s fast and easy to build code-free or code-centric ETL and ELT processes. In this scenario, learn how to create code-free pipelines within an intuitive visual environment.
In today’s data-driven world, big data processing is a critical task for every organisation. To unlock transformational insights, data engineers need services that are built to simplify ETL as well as handle the complexities and scale challenges of big data integration. With Azure Data Factory, it’s fast and easy to build code-free or code-centric ETL and ELT processes.
Simplify ETL at scale
Maria, a data engineer, receives a never-ending stream of requests to bring in more data from different data sources into her company’s reports. For each new data source, Maria must research, build, connect and manage the integration, which is incredibly time-consuming.
Maria is convinced that her team needs a more scalable approach, and looks to Azure Data Factory to start the journey towards a Modern Data Warehouse.
Maria, a data engineer, receives a never-ending stream of requests to bring in more data from different data sources into her company’s reports. For each new data source, Maria must research, build, connect and manage the integration, which is incredibly time-consuming.
Maria is convinced that her team needs a more scalable approach, and looks to Azure Data Factory to start the journey towards a Modern Data Warehouse.
Use built-in data connectors
Maria links her environment to Amazon S3 to retrieve customer data. S3 is just one of 90+ built-in connectors available in Azure Data Factory.
Copy data to the data lake
In her pipeline, she adds a Copy activity and selects the source data store as S3. She can now preview the data before running the job. It looks like it will need some work to clean it up and align with organisational standards.
Sample and review data
She then selects her Azure Data Lake as the sink data store
Visually explore and refine data
By adding a wrangling data flow to her pipeline, Maria can now start to prepare her dataset. She can easily generate summary statistics and apply steps to remove broken rows and fix columns.
Protect against changes to upstream schemas
She now uses a mapping data flow to complete the transformation. She enables “Allow schema drift” on the input to improve resilience to upstream changes.
Transform and aggregate data without code
She adds a join operator to add in transactional data already in the lake, and then aggregates the values to get metrics such as per customer spend. Finally, she lands this data in Azure Synapse Analytics, where the data will be analysed to unlock limitless insights.
Trigger pipelines on schedule
With the pipeline complete, she can schedule it for execution on a recurring basis, on a tumbling window or on the detection of new files.
Manage pipelines at scale
The run history of her pipeline will be visible alongside all of the other pipelines in her organisation.
Monitor and debug visually
For each run, Maria gets a real-time, visual representation of progress. She can observe the progress of each stage, making monitoring and debugging easy.
Bring DevOps to your data
Now that the pipeline is complete, she publishes her changes to Git with a single click. The work she’s done is now under version control and can be part of her team’s CI/CD workflow.
Integrate all of your data, code-free
With Azure Data Factory, Maria has been able to ingest, transform and operationalise the integration of a new data source without having to write a single line of code. With this new data in the data warehouse, her entire organisation can start exploring it using self-service tools such as Power BI, resulting in better data-driven decisions across the organisation.
Try it yourself with our hands on lab Create your Azure free account today
With Azure Data Factory, Maria has been able to ingest, transform and operationalise the integration of a new data source without having to write a single line of code. With this new data in the data warehouse, her entire organisation can start exploring it using self-service tools such as Power BI, resulting in better data-driven decisions across the organisation.
Try it yourself with our hands on lab Create your Azure free account today
Trusted global cloud presence
- Access Data Factory in more than 25 regions globally to ensure data compliance, efficiency and reduced network egress costs.
- Data Factory has been certified by HIPAA and HITECH, ISO/IEC 27001, ISO/IEC 27018 and CSA STAR.
- Connect securely to Azure data services with managed identity and service principal. Store your credentials with Azure Key Vault.
- Managed virtual network gives you an isolated and highly secure environment to run your data integration pipelines.
Only pay for what you need, with no upfront cost
Explore a range of cloud data integration capabilities to fit your scale, infrastructure, compatibility, performance and budget needs. Options include managed SSIS for seamless migration of SQL Server projects to the cloud and large-scale, serverless data pipelines for integrating data of all shapes and sizes.
Data Factory PricingAzure Data Factory resources and documentation
Get started with learning resources
Trusted by companies of all sizes
Data Factory helps Adobe deliver data-rich digital experiences at scale
Nik Shroff, Director of Microsoft Solutions at Adobe"We want to focus on ways to help our customers ingest data, clean it, store it at scale, and access it globally in real time. With Azure, we can rely on our own core competencies, and not have to build the underlying infrastructure."
Analytics firm cuts data warehouse dev time by 80 per cent
With just a few clicks, customers can use Concentra’s SaaS data solutions to create an end-to-end data management process through SSIS ETL workloads in a managed SSIS environment within Data Factory, achieving high scalability and availability while reducing operational costs.
Azure Data Factory updates, blogs and announcements
-
UPDATE
Azure Data Factory data flow connectors now generally available for CDM and Delta Lake.
-
UPDATE
Azure Data Factory is now available in five new regions
-
UPDATE
Azure Data Factory adds cached lookups and enhanced zoom design to data flows
-
UPDATE
GA: Data Factory adds ORC data lake file format support for ADF Data Flows and Synapse Data Flows
-
UPDATE
Public Preview: Data Factory adds SQL Managed Instance (SQL MI) support for ADF Data Flows and Synapse Data Flows
-
UPDATE
Update .NET to 4.7.2 for Azure Data Factory upgrade by 01 Dec 2020
-
UPDATE
Migrate your Azure Data Factory version 1 to 2 service
-
UPDATE
Data Factory SQL Server Integration Services (SSIS) migration accelerators are now generally available
Frequently asked questions about Data Factory
-
Data Factory is available in more than 25 regions.
-
We guarantee that we will successfully process requests to perform operations against Data Factory resources at least 99.9 per cent of the time. We also guarantee that all activity runs will be initiated within four minutes of their scheduled execution times at least 99.9 per cent of the time. Read the full Data Factory service-level agreement (SLA).
-
Integration runtime (IR) is the compute infrastructure that Data Factory uses to provide data integration capabilities across network environments. IR moves data between the source and destination data stores by providing scalable data transfer. IR executes Data Flow authored visually in a scalable way on Spark compute runtime. IR provides the capability to natively execute SSIS packages in a managed Azure compute environment. It supports dispatching and monitoring of transformation activities running on several compute services. For more information, see integration runtime in Data Factory.