{"id":2056,"date":"2018-10-25T00:00:00","date_gmt":"2018-10-25T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/azure-cosmos-db-partitioning-design-patterns-part-1"},"modified":"2023-05-11T15:36:24","modified_gmt":"2023-05-11T22:36:24","slug":"azure-cosmos-db-partitioning-design-patterns-part-1","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/azure-cosmos-db-partitioning-design-patterns-part-1\/","title":{"rendered":"Azure Cosmos DB partitioning design patterns \u2013 Part 1"},"content":{"rendered":"

In this article, you will learn how to use partition keys to efficiently distribute data, improve application performance, and enable faster look-up. The pre-requisites of this article are general knowledge of Azure Cosmos DB and a good understanding of change feed<\/a>, request unit<\/a> (RU), and Azure Functions<\/a>.<\/p>\n

Imagine you have data which you would like to insert with high throughput and query on two or more different keys. In this scenario, suppose you work for an airline company and need to store user reservation information in a collection. User data is defined as:<\/p>\n

\r\n{\r\n     UserId: user@email.com,\r\n     FirstName: John,\r\n     LastName: Doe,\r\n     PNR: 2345423,\r\n     CityOfOrigin: Seattle,\r\n     CityOfDestination: London,\r\n     DOB: 12.2.76,\r\n     other details \u2026.\r\n}<\/pre>\n
Out of many possible values, you choose \u201cUserId\u201d (user email address) as the partition key. This is a good choice for a partition key because \u201cUserId\u201d is unique for every user, ensuring your data remains well distributed. Your data is distributed evenly among all partitions, as shown in Figure 1. However, when you are querying the data you do not always have a \u201cUserId\u201d. Sometimes you want to query the data by user last name or user Passenger Name Record (PNR) number.<\/p>\n
\"GoodPartition\"<\/p>\n
Figure 1: Data distributed evenly across partitions<\/em><\/p>\n
Azure Cosmos DB indexes all data by default. If you try to query the data by “LastName”, you will get the result, but it will cost you more request units<\/a> (RU\/s) because queries without partition key become fan-out queries. Fan-out queries check all partitions, which will cost you extra RU\/s and may affect the performance of your application. If you have a small number of partitions with less data, you may not perceive any significant side effects of fan-out queries, but when you start getting in high numbers of partitions and large amounts of data, fan-out queries can be detrimental to your applications. Infrequent cross partition query is fine, but if it is a frequent query, then what is the solution?<\/p>\n
One option is to have two more lookup collections PNR and \u201cLastName\u201d for the mapping of PNR to \u201cUserId\u201d, and \u201cLastName\u201d to \u201cUserId\u201d. The PNR collection will have PNR as the partition key and row key and \u201cUserId\u201d as the value.\u00a0<\/p>\n
\"clip_image002\"<\/p>\n
These different lookup collections will make your application more efficient. To lookup detail by PNR, first query the PNR collection to get the \u201cUserId\u201d. Then use \u201cUserId\u201d to query user collection for all details. These two calls can complete within a few milliseconds and will consume fewer RU\/s than a single fan-out query. Most of the point lookup can complete within one to two milliseconds. Even after two lookups, you can be done within 10 milliseconds for most of the queries.<\/p>\n
You may not want a few extra milliseconds added to calls and instead may decide to duplicate the data in the PNR and \u201cLastName\u201d collections. While this will enable fast look-up, it isn\u2019t recommended because it may add complexity and costs when data is updated. Ultimately, you must balance requirements, performance, and complexity. Starting with a simple solution is often the best approach.<\/p>\n
Now let\u2019s look at the distribution of your data in different collections. For example, if you look at the \u201cLastName\u201d collection you will find that the data is not equally distributed because there are more people with the last name \u201cSmith\u201d than \u201cZubrkee\u201d. In this case, the data will look very much like Figure 2.<\/p>\n
\"BadPartition\"<\/p>\n
Figure 2: Data unevenly distributed across partitions<\/em><\/p>\n
The data in this scenario is distributed unevenly, with some full partitions and others greatly underused. This type of data distribution creates many problems.<\/p>\n