New Common Data Model connector for Apache Spark in Azure Synapse Analytics and Azure Databricks (in preview)
Published date: 30 September, 2020
The Common Data Model (CDM) provides a consistent way to describe the schema and semantics of data stored in Azure Data Lake Storage (ADLS). This enables data to be exported in CDM format from applications such as Dynamics 365 and easily mapped to the schema and semantics of data stored in other services.
Today we are announcing a new CDM connector that extends the CDM ecosystem by enabling services that use Apache Spark to now read and write CDM-described data in CSV or Parquet format. This is done through a data frame abstraction that can be accessed from Scala, Python or Spark SQL. This new Spark CDM connector requires zero configuration and is pre-installed with Azure Synapse Analytics. It can also be installed and used with Azure Databricks.