Apache Spark 3.0 is now available for public preview in HDInsight 4.0.
Published date: 14 April, 2021
Apache Spark 3.0 marks a major release from version 2.x and introduces significant improvements over previous releases. Apache Spark 3.0 adds performance features such as Adaptive Query Execution (AQE) and Dynamic Partition Pruning (DPP) along with improvements for ANSI SQL by adding support for new built-in functions, additional Join hints and DML operators such as DELETE, UPDATE and MERGE.
For a complete list of improvements, see the Apache Spark 3.0 release notes.
The Apache Spark 3.0 public preview is being deployed across all Azure regions where HDInsight is available. Please reach out to askhdinsight@microsoft.com if you have questions about availability in your favourite region.