Public preview: Azure Synapse Analytics Spark elastic pool storage
Published date: 28 July, 2022
Azure Synapse Analytics Spark pools now support elastic pool storage. Apache Spark in Azure Synapse Analytics utilise temporary disk storage while the Spark pool is instantiated. For many Spark jobs, it’s difficult to estimate the pools storage requirements, which can cause some Spark jobs to fail if the temporary storage is exhausted. Today we are enabling elastic pool storage which allows the Spark engine to monitor worker node temporary storage and attach additional disks if needed. No action is required by you, and you should see fewer job failures as a result.