Data Factory adds new hierarchical data handling and new flexibility for complex joins
Published date: 01 May, 2020
Our most recent updates to Data Factory include:
- Hierarchical data modeling in data flows has been expanded with Flatten, collect aggregation, and updated sink mapping. We've also published a new tutorial on how to use these capabilities when moving from relational to hierarchical with Azure SQL DB and CosmosDB.
- Non-equi join support added to Join, Lookup, and Exists transformations will land in the coming weeks. You will now be able to add non-equi joins to all ADF join types in the UI and add "and" conditions with additional equi/non-equi types.
- An update that already landed in ADF for Joins, Exists, and Lookups is an update to the Optimize tab. You can now switch off native Spark data push-down to execution nodes when performing a join using the broadcast option. The default is auto and you can use "fixed" override the default data flow behavior in Spark by broadcasting only left or right side.
Read the blog to learn more.