Generally available: Azure Databricks SQL Pro
Published date: 01 November, 2022
Beginning in November, Databricks is rolling out a new compute option called Databricks SQL Pro, joining the SQL product family of Classic and Serverless. Like Serverless SQL, SQL Pro includes performance and integration features that expand the SQL experience on the Lakehouse Platform. The primary difference is that SQL Pro keeps compute in the customer’s account.
Azure Databricks SQL Pro’s features include:
- Predictive I/O – Predictive I/O boosts query performance by leveraging the years of experience Databricks has in building large AI/ML systems to make the lakehouse a more intelligent data warehouse
- Applying deep learning techniques to our data lakehouse
- Determining the most efficient access pattern to read data
- For select queries, Predictive I/O calculates probabilities to anticipate where the next row match will occur, only reading that data from cloud storage
- Native Geospatial Features – 30+ built-in H3 expressions for geospatial processing and analysis in Photon-enabled clusters, available in SQL, Scala and Python
- Query federation – Databricks Warehouse now supports the ability to query live data from various databases through federation capability. Query federation allows BI applications to integrate data in the lakehouse and external data sources, providing a rich query experience
- Workflows integrations – Invoke DBSQL tasks, such as queries, alerts, and running dashboards, from within Databricks Workflows
Many more feature and performance improvements are on the way, such as Materialised Views and Python UDFs. SQL Pro is generally available everywhere Databricks SQL Classic is available.