We are glad to announce that Azure Data Factory has added more new features in July, including:
- Preview for Data Management Gateway high availability and scalability
- Skipping or logging incompatible rows during copy for fault tolerance
- Service principal authentication support for Azure Data Lake Analytics
We will go through each of these new features one by one in this blog post.
Preview for Data Management Gateway high availability and scalability
You can now associate multiple data management gateway nodes that are installed on different machines with a single logical gateway, so as to avoid Data Management Gateway being the single point of failure. In addition, this helps to scale out to achieve better copy performance. You can also choose to scale up each gateway node based on your load. Moreover, Azure Data Factory now provides a richer monitoring experience on gateway status and resource utilization from Azure portal. Learn more from Data Management Gateway – High Availability and Scalability Preview.
Skipping or logging incompatible rows during copy for fault tolerance
When copying data using Azure Data Factory Copy Activity, you now have different options to deal with incompatible data between source and sink data stores. You can choose to either abort and fail the copy run upon encountering incompatible data (default behavior), or continue copying all the data by skipping those incompatible rows. Additionally, you also have the option to log the incompatible rows in Azure Blob so you can examine the cause for failure, fix the data on the data source and retry. The feature is available via both Copy Wizard and JSON editing. Learn more about supported scenarios and configuration from the documentation page, Copy Activity fault tolerance – skip incompatible rows.
Service principal authentication support for Azure Data Lake Analytics
To use U-SQL Activity, Azure Data Factory now supports service principal authentication for Azure Data Lake Analytics like we did for Azure Data Lake Store earlier, in addition to the existing user credential authentication. We recommend that you use service principal authentication to get rid of periodical token expiration behavior with user credentials, especially for a scheduled U-SQL execution. Learn more about supported authentication types and configuration from documentation, Transform data by running U-SQL scripts on Azure Data Lake Analytics.
Above are the new features we introduced in July. Do you have questions or feedback? Share your thoughts with us on Azure Data Factory forum or feedback site, we’d love to hear more from you.