One year of Azure Cosmos DB
Today marks the 1-year anniversary of the launch of Azure Cosmos DB – Microsoft’s globally distributed, multi-model database service. As many of you know, Azure Cosmos DB started inside Microsoft in 2010 as Project Florence with a goal to address the unmet needs of Microsoft’s own globally distributed apps. These apps were pushing the limits of traditional databases and were unable to achieve desired availability, scalability and performance needs at global scale. This cemented our “mission” to build a database for the cloud from the ground up to enable globally-distributed apps.
Reflecting back on the past year, it is deeply heartening for us to see the growth of the service. We are so very grateful to our customers who have relied on Azure Cosmos DB to build their mission-critical apps, pushed the limits of the service and always demanded the best. To all our customers – we want to thank you for trusting us with your data. We are committed to making Azure Cosmos DB the best database for your business.
As a cloud-born database, we have carefully engineered Azure Cosmos DB with multi-tenancy and global distribution from the ground up. Azure Cosmos DB provides, turnkey global distribution, elastic scaling of throughput and storage worldwide, single-digit millisecond latencies at the 99th percentile, five well-defined consistency models, and guaranteed high availability, all backed by industry-leading comprehensive SLAs. Azure Cosmos DB is completely schema-agnostic and automatically indexes all data. This means that you don’t have to deal with cumbersome schema changes, version management and manage indexes across multiple regions. Finally, thanks to the wire-compatible APIs for MongoDB/Apache Cassandra/Apache Gremlin on top of a write-optimized, multi-model database engine, migrating data from IaaS hosted/on-premises NoSQL databases to Azure Cosmos DB is more friction-free than before.
Over the past year, we have delivered new innovative capabilities, week after week – from the new consistency models to serverless, from industry-leading, comprehensive SLAs to native Spark integration for enabling operational analytics at global scale. The new Cosmos DB capabilities which we are announcing at //Build 2018 is another crucial step towards enabling you to easily write globally distributed apps for the Intelligent-Cloud-Intelligent Edge era.
Azure Cosmos DB at //Build 2018
Today at the Microsoft Build Conference 2018 in Seattle, we are excited to announce several new capabilities all of which are intended to enable you to easily build your mission-critical, globally distributed apps. The following are notable announcements:
Unprecedented write scalability and availability with multi-master replication (Preview): With Azure Cosmos DB, we have harvested decades of distributed systems research, married with rigorous engineering and operationalized it at a massive scale. For Cosmos DB’s multi-master replication protocol, we have (a) applied several key ideas from the research work done on Bayou, (b) integrated conflict-free-replicated-data types (CRDTs) natively with the core type system of Cosmos DB’s database engine, and (c) verified the correctness properties of our protocol with TLA+, so we can continue to offer multiple, well-defined consistency models and industry leading SLAs. Cosmos DB’s multi-master replication protocol, in-turn, enables the following capabilities:
- Unlimited and elastic write scalability, all around the world. With Azure Cosmos DB, you can now elastically scale the writes (in addition to the reads) all around the world, across any number of Azure regions, any time.
- 99.999% write availability, all around the world. Azure Cosmos DB offers 99.999% write availability (in addition to 99.999% read availability) all around the world, backed by the industry-leading SLAs.
- Guaranteed low write latency at the 99th percentile, all around the world. In addition to the guaranteed single-digit-millisecond read latencies, Azure Cosmos DB now delivers a single digit write latency at the 99th percentile, anywhere around the world, which is guaranteed by the financially-backed SLAs.
- Unlimited endpoint scalability, all around the world. Azure Cosmos DB’s replication protocol is designed to support unlimited number of endpoints – from 100s of cloud regions to billions of edge devices. Its master-less replication protocol is designed to treat cloud regions and edge endpoints homogeneously – cloud regions and edge devices are true peers. We have architected Azure Cosmos DB to enable ISVs and developers to write occasionally connected edge apps and seamlessly connect databases running on-premises and on other clouds to Cosmos DB in an active-active manner.
- Multiple well-defined consistency models to choose from. Azure Cosmos DB’s multi-master replication protocol composes extremely well with various consistency models it already exposes. You continue to get an intuitive programming model for data consistency with clear tradeoffs for availability, latency and throughput for a given consistency model.
- Intuitive and flexible programming model for conflict management. In master-less database architectures, write-write conflicts are possible. In a typical master-less system, detection and resolution of conflicts are usually left as burden for application developers. The design of Azure Cosmos DB’s replication protocol ensures conflicts are automatically detected by the system. We also provide multiple, well-defined conflict resolution modes that guarantees strong convergence.
To demonstrate Azure Cosmos DB’s new multi-master replication capability, we have built a PxDraw (https://pxdraw.azure.com), a massively scalable, multi-user, real-time drawing canvas, distributed across 22+ regions across the world, leveraging multi-master capability to provide guaranteed low write latency. Join us today to draw on the canvas, one pixel at a time, in unison with thousands of other people from around the world! We begin the drawing session now and will end it after the conference has wrapped up on Wednesday, May 9th. You can read more about it here.
General availability of VNET Service Endpoint: We are excited to announce general availability of VNET Service Endpoint. VNET Service Endpoint enables Azure customers to directly communicate with Azure Cosmos DB from their Virtual Networks without compromising security by restricting access of database to customer owned VNET. You can now create network rules that only allow traffic from selected virtual networks and subnets. This feature is now available in all regions of Azure public cloud. You can now combine existing authorization mechanisms like Firewall Access Control List (ACL) with the new network boundaries to provide an enhanced security for their data. Azure Cosmos DB is the first service to allow cross-region access control support, where you can restrict access to a globally distributed Azure Cosmos DB account from subnets located in multiple regions. Read more about it here.
Provisioning throughput on Cosmos DB database (Preview): So far, you could only provision throughput for a Cosmos DB container (which could be a collection, a table or a graph – depending on the data model and API you are using) either programmatically or via Azure portal. Now, you can also configure throughput on your Cosmos DB database. Provisioning throughput on a Cosmos DB database, allows you to share the throughput among all the containers that belong to that database. This allows you to “pool” the provisioned throughput across a set of containers within a Cosmos DB database. In fact, within a given Cosmos DB database, you can mix and match containers which “share” the throughput among themselves as well as specific containers which have been assigned dedicated throughput. By provisioning throughput at different granularities (e.g., containers and databases), you can now better optimize your costs based on your workloads’ characteristics.
If your container or a Cosmos DB database are distributed across multiple regions, then the throughput you configure on the container or a database is guaranteed to be made available across all regions. Read more about it here.
More Cosmic developer goodness!
- General availability of Bulk Executor library for Azure Cosmos DB: We are excited to announce general availability of the Azure Cosmos DB BulkExecutor library. The BulkExecutor library is designed as an extension to Cosmos DB’s core SDKs and provides out-of-the box functionality to perform bulk operations in Cosmos DB. On average, BulkExecutor library delivers >10x improvement for write throughput. This library also provides out-of-the-box efficient handling of rate-limiting, timeouts and transient exceptions. It allows easier scale-out by adding additional BulkExecutor client instances on individual VMs to achieve even greater write scalability. You can download the .NET version of BulkExecutor library via Nuget and the Java version via Maven. API documentation, performance tips and sample apps to get you started can be found here for .NET and here for Java. Read more about it here.
- Asynchronous Java SDK for Azure Cosmos DB: We are excited to announce the general availability of a new asynchronous Java SDK for Cosmos DB’s SQL API. This SDK leverages the popular RxJava library to add a new async API surface area for composing event-based programs with observable sequences. It also features an improved user experience and offers 2x client-side performance improvement!
Customer momentum
Today, some of the world’s largest internet-scale businesses such as Rolls-Royce, Jet.com, Dominos, ASOS and countless other enterprises depend on Cosmos DB for their mission-critical workloads. Jet.com, for instance, uses Azure Cosmos DB for multiple retail scenarios, including order processing, inventory management and pricing. Jet.com also leverage micro-services with Azure Cosmos DB to store their respective events and process them using event-sourced architecture. Jet.com uses Change Feed API to process events in the committed order and to communicate across multiple services that are globally distributed to cater to the broad data footprint and latency needs. Dominos uses Azure Cosmos DB to represent the shopping cart state to expedite the order delivery and sharing the various retail insights across different franchises across the globe. Dominos relies heavily on Cosmos DB’s reliable global distribution capabilities, low latency and strong enterprise grade SLAs. Within Microsoft, Cosmos DB has become ubiquitous — Microsoft’s own mission critical applications and services including LinkedIn, Skype, XBOX, Active Directory, Azure Portal, Office 365, Bing, Universal Store and countless others all rely on Cosmos DB for managing their data globally, at scale.
Our Customers have told us the transformative role Azure Cosmos DB has played in their businesses. Following are a few quotes from some of them:
“At ASOS, we experience tremendous customer demand handling >100M unique daily visitors across >200 countries shopping over >100K of products. We needed a distributed database that seamlessly and elastically scales to serve our global customers. During peak sales periods we need extra computing power to cope with customer demand and manage our large datasets automatically behind the scenes without the burden of complex online data migrations – freeing our engineering teams to concentrate on business features, reducing the need for complex coordination with business + operations teams, and most importantly, seamlessly scaling with zero planned downtime. Well, I have to say I’ve been impressed with Azure Cosmos DB – we’ve watched our collections seamlessly grow as our applications have grown while maintaining its 99.999% availability SLA.”
– Gary Strange, Lead Data Engineer, ASOS
“Data at Jet.com and Walmart Labs is scale. E-commerce will be a $1 trillion business in the next five to ten years, and Azure Cosmos DB is critical to our vision, because we need a database that will elastically scale with us. Not only that – but we also need true elasticity and the ability to scale up/down on demand to help us manage our costs carefully. We experience big peaks during key shopping days, where expected rates of events can increase 10 to 20 compared to the rest of the year.
For example, for Black Friday and Cyber Monday – we needed the ability to enough provisioned throughput to satisfy 1 trillion Request Units over 24 hours to satisfy our customer demand. To solve this – we deployed a geo-replicated Cosmos DB collection configured for 10 million request units per second.
Another key aspect for us is performance. We have rigorous latency objectives to ensure a responsive customer experience – in which we really like Cosmos DB’s predictable latency characteristics where we observe single-digit millisecond latency reads and writes.
Cosmos DB’s elastic scale, automatic indexing and Change Feed support makes it an excellent event store for event sourcing. Heterogeneous events can be automatically indexed for query, and the event log exposed through Change Feed can be used to reconstruct past states. Performing writes in an append-only manner via event sourcing is fast and scalable for systems that are write-heavy, in particular when used in combination with a write-optimized database. Many of our microservices are mission critical components to our business. Cosmos DB gives us peace of mind with an aggressive 99.99% availability guarantee when deployed to a single region – which is further strengthened to 99.999% when deployed across regions using Cosmos DB’s global distribution.”
– Scott Havens, Director of Engineering, Jet.com and Walmart Labs
“We chose Azure Cosmos DB as the data storage solution for our DigitalVault cloud-based platform. Johnson Controls is a global company, with a presence on all seven continents. Having a global database like Azure Cosmos DB available makes it dramatically easier to build applications to support our customers and buildings wherever they may be. Cosmos DB gives us the low latency and solid SLAs we need, and with fine-grained control over consistency we can make the right choices for our application for performance.
We leverage the indexing and various APIs and data models that Azure Cosmos DB offers. Using the data and Azure Cosmos DB to create a virtual version of a physical structure, such as a floor or room, making it easy to conceptualize the relationships between building assets. Azure Cosmos Gremlin API and interfaces greatly expedited the development and integration process to allow us to maximize development capabilities and minimize integration time.”
– Ada Ma, Head of Platform and Partnership, Johnson Controls
“Azure Cosmos DB frees our developers to focus on delivering value to our MyQ users by providing unlimited scale and global distribution in a fully managed service.”
– Jeff Paine, MyQ Middleware Manager, Chamberlain Group
“We see great potential for Azure Cosmos DB in our products…. Building a flexible, scalable data layer with Azure Cosmos DB will enable us to deliver actionable insights to our users.”
– Kaustubh Page, Senior Product Manager for Analytical Insights and Design Insights, Bentley Systems
“Azure Cosmos DB has enabled us to put our graph data where our customers are, globally. Combining this flexibility with the capability to store documents, graph objects, and tabular data, all in the same place, has meant we are saving time developing our applications.”
– Dave McHugh, Chief Technology and Data Architect, JATO Dynamics
“Archive2Azure’s tight integration between Azure Cosmos DB Gremlin API and Azure Blob Storage is a major pillar that supports our intelligent information management platform at scale and across dozens of different file types.”
– Tibi Popp, CTO, Archive360
“As a developer, having access to a cloud service that takes just a few hours to hook up is gold. Azure Cosmos DB provides so much right out of the box and seems to have an answer for all our needs, like configurable TTL, encryption, RESTful API, and speed—oh, is it fast! These are all features we did not have to implement so we could instead focus on the business layer of our application. Azure Cosmos DB and the other Azure PaaS services allowed us to ramp up our development cycles, which had a multiplicative effect. The best part is, Azure Cosmos DB continues to evolve; I can’t wait to unlock its full potential.”
– Jose Molina-Melendez, Software Engineer at Allscripts
What are you going to build with Azure Cosmos DB next?
We are delighted to see the continued amazing growth of Cosmos DB since its launch. We are thankful to our customers and are committed to delivering the best globally distributed database. With the new capabilities which we are announcing at //Build 2018, we believe that you will be able to write amazingly powerful, globally distributed apps. We want to help you to truly transform the world through the apps you build on Azure Cosmos DB.
If you are using Azure Cosmos DB, please feel free to reach out to us at AskCosmosDB@microsoft.com any time. If you are not yet using Azure Cosmos DB, you can try Azure Cosmos DB for free today, no sign up or credit card is required. If you need any help or have questions or feedback, please reach out to us any time. For the latest Azure Cosmos DB news and features, please stay up-to-date by following us on Twitter #CosmosDB, @AzureCosmosDB. We look forward to see what you will build with Azure Cosmos DB!
– Your friends at Azure Cosmos DB