Azure Databricks pricing

A Databricks unit (“DBU”) is a unit of processing capability per hour, billed on per-second usage.

The Data Engineering workload is defined as a job that both starts and terminates the cluster on which it runs. For example, a workload may be triggered by the Databricks job scheduler, which launches a new Apache Spark cluster solely for the job and automatically terminates the cluster after the job is complete.

The Data Analytics workload is any workload that is not an automated workload, for example, running a command within Databricks notebooks. These commands run on Apache Spark clusters which may persist until manually terminated. Multiple users can share a cluster to perform interactive analysis collaboratively.

Depending on the type of workload your cluster runs, you will either be charged for Data Engineering or Data Analytics workload. For example, if the cluster runs workloads triggered by the Databricks jobs scheduler, you will be charged for the Data Engineering workload. If your cluster runs interactive features such as ad-hoc commands, you will be billed for Data Analytics workload.

• If you run Premium tier cluster for 100 hours in East US 2 with 10 DS13v2 instances, the billing would be the following for Data Analytics workload:
  • VM cost for 10 DS13v2 instances —100 hours x 10 instances x $0.598/hour = $598
  • DBU cost for Data Analytics workload for 10 DS13v2 instances —100 hours x 10 instances x 2 DBU per node x $0.55/DBU = $1100
  • The total cost would therefore be $598 (VM Cost) + $1100 (DBU Cost) = $1698.
• If you run Premium tier cluster for 100 hours in East US 2 with 10 DS13v2 instances, the billing would be the following for Data Engineering workload:
  • VM cost for 10 DS13v2 instances —100 hours x 10 instances x $0.598/hour = $598
  • DBU cost for Data Engineering workload for 10 DS13v2 instances —100 hours x 10 instances x 2 DBU per node x $0.35/DBU = $700
  • The total cost would therefore be $598 (VM Cost) + $700 (DBU Cost) = $1298.

In addition to VM and DBU charges, you may also be charged for managed disks, public IP address or any other resource such as Azure Storage, Azure Cosmos DB depending on your application.

We charge for the number of minutes your cluster is running rounded to the nearest minute, not hour.

High concurrency pools are available on Azure Databricks for the Data Analytics workload only. High concurrency pools are available on Standard and Premium tiers. A High concurrency pool consists of worker nodes and one driver node. Customers can pick any available VM instance as worker nodes for their high concurrency pool. In addition, customers will have to specify the minimum and maximum number of workers for each high concurrency pool. All worker nodes in a pool must be the same VM instance. Depending on the load on the high concurrency pool, Azure Databricks will automatically scale the number of instances in a pool within the specified minimum and maximum worker nodes. On a per minute basis, customers will be billed for each VM running in the pool and the corresponding DBU’s for those VM instances. In addition, the service will pick a VM instance for one driver node for the high concurrency pool. Customers will be billed for the VM and the corresponding DBU charges for the driver node.

Let us say you choose a Premium tier high concurrency pool of DS13v2 instances (Azure Databricks charges 2 DBU’s per hour for this VM instance) and you set the minimum and maximum worker nodes to be 2 and 10 respectively. Let us assume this high concurrency pool runs for 3 hours. Let us also assume that based on the load, Azure Databricks runs 2 worker nodes for the first hour, 8 worker nodes for the second hour and 3 worker nodes for the third hour, before you shut the high concurrency pool down. In addition, Azure Databricks will pick a driver node that will be billed through the life time of the high concurrency pool. Let us say Databricks picks D13v2 for this example.

Your bill for the high concurrency pool looks as follows: