Skip to main content

Generally available: Apache Spark 3.1.2 in Azure HDinsight

Published date: April 04, 2022

The latest release of Apache Spark 3.1 includes:

  • Adaptive query execution
  • Convert sort merge join to broadcast hash join
  • Spark catalyst optimizer
  • Dynamic partition pruning
  • Create new Spark 3.1 clusters (not Spark 3.0 clusters (preview))

For additional details, review the document Apache Spark 3.1 on Microsoft Tech Community.

For a complete list of improvements, review the Apache Spark 3.1 release notes.

For more details on migration, review the migration guide.

  • Open Source