Public preview: Intelligent cache for Apache Spark in Azure Synapse
Published date: 24 March, 2022
Decrease processing time and increase performance on repeat queries with the new intelligent cache for Apache Spark in Azure Synapse. Unlike Apache Spark’s native caching feature, the intelligent cache automatically stores each read within the allocated cache storage space which allows detection of underlying file changes and refreshes the files providing the most recent data.
The intelligent cache works seamlessly behind the scenes and caches data to help speed up the execution of Apache Spark as it reads from your ADLS Gen2 data lake. It also automatically detects changes to the underlying files and will automatically refresh the files in the cache, providing you with the most recent data. When the cache size reaches its limit, the cache will automatically release the least read data to make space for more recent data.
To learn more, explore our documentation.