Generally available: Apache Spark 3.1.2 in Azure HDinsight

The latest release of Apache Spark 3.1 includes:

• Adaptive query execution
• Convert sort merge join to broadcast hash join
• Spark catalyst optimizer
• Dynamic partition pruning
• Create new Spark 3.1 clusters (not Spark 3.0 clusters (preview))

For additional details, review the document Apache Spark 3.1 on Microsoft Tech Community.

For a complete list of improvements, review the Apache Spark 3.1 release notes.

For more details on migration, review the migration guide.