Hello VLDB attendees!<\/p>\n
Welcome to Rio De Janeiro. I wanted to take this opportunity to share with you some of the exciting work in data that\u2019s going on in the Azure Data team at Microsoft, and to invite you to take a closer look.<\/p>\n
Microsoft has long been a leader in database management with SQL Server, recognized as the top DBMS by Gartner for the past three years in a row. The emergence of the cloud and edge as the new frontiers for computing, and thus data management, is an exciting direction\u2014data is now dispersed within and beyond the enterprise, on-premises, on-cloud, and on edge devices, and we must enable intelligent analysis, transactions, and responsible governance for all data everywhere, from the moment it is created to the moment it is deleted, through the entire life-cycle of ingestion, updates, exploration, data prep, analysis, serving, and archival.<\/p>\n
These trends require us to fundamentally re-think data management. Transactional replication can span continents. Data is not just relational. Interactive, real-time, and streaming applications with enterprise level SLAs are becoming common. Machine learning is a foundational analytic task and must be supported while ensuring that all data governance policies are enforced, taking full advantage of advances in deep learning and hardware acceleration via FPGAs and GPUs. Microsoft is a very data-driven company, and to support product teams such as Bing, Office, Skype, Windows, and Xbox, the Azure Data team operates some of the world\u2019s largest data services for our internal customers. This grounds our thinking about how data management is changing, through close collaborations with some of the most sophisticated application developers on the planet. Of course, even in a company like Microsoft, the majority of data owners and users are domain experts, not database experts. This means that even though the underlying complexity of data management is growing rapidly, we need to greatly simplify how users deal with their data. Indeed, a large cloud database service contains millions of customer databases, and requires us to handle many of the tasks previously handled by database administrators.<\/p>\n
We are excited about the opportunity to re-imagine data management. We are making broad and deep investments in SQL Server and open source technologies, in Azure data services that leverage them, as well as in new built-for-cloud technologies. We have an ambitious vision of the future of data management that embraces Data Lakes and NoSQL, on-premises, in the cloud, and on edge devices. There has never been a better time to be part of database systems innovation at Microsoft, and we invite you to explore the opportunities to be part of our team.<\/p>\n
This is an exciting time in our industry with many companies competing for talent and customers, and as you consider options, we want to highlight how Microsoft is differentiated in several respects. First, we have a unique combination of a world-class data management ecosystem in SQL Server and a leading public cloud in Azure. Second, our culture puts customers first with a commitment to bringing them the best of open source technologies alongside the best of Microsoft. Third, we have a deep commitment to innovation. Product teams collaborate closely with research and advanced development groups at Cloud Information Services Lab, Gray Systems Lab, and Microsoft Research to go farther faster, and to maintain strong ties with the research community.<\/p>\n
In this blog, I\u2019ve listed the many services and ongoing work in the Azure Data group at Microsoft, together with links that will give you a closer look. I hope you will find these of interest.<\/p>\n
Have a great VLDB conference!<\/p>\n
Rohan Kumar<\/p>\n
Corporate Vice President, Azure Data<\/strong> The Microsoft data platform continues momentum for cloud-scale innovation and migration<\/a><\/p>\n Azure data blog posts by Rohan Kumar<\/a><\/p>\n Industry leading database management system, now available on Linux\/Docker and Windows, on-premises and in the cloud.<\/p>\n The SQL Server blog<\/a><\/p>\n SQL Server 2017<\/a><\/p>\n Azure Cosmos DB is the first globally-distributed data service that lets you elastically scale throughput and storage across any number of geographical regions while guaranteeing low latency, high availability, and consistency \u2013 backed by the most comprehensive SLAs in the industry. Azure Cosmos DB is built to power today\u2019s IoT and mobile apps, and tomorrow\u2019s AI-hungry future.<\/p>\n It is the first cloud database to natively support a multitude of data models and popular query APIs, is built on a novel database engine capable of ingesting sustained volumes of data, and provides blazing-fast queries \u2013 all without having to deal with schema or index management. And it is the first cloud database to offer five well-defined consistency models, so you can choose just the right one for your app.<\/p>\n Azure Cosmos DB: The industry's first globally-distributed, multi-model database service<\/a><\/p>\n Easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python, and .NET over petabytes of data. With no infrastructure to manage, you can process data on demand, scale instantly, and only pay per job. There is no infrastructure to worry about because there are no servers, virtual machines, or clusters to wait for, manage, or tune. Instantly scale the processing power for any job.<\/p>\n Azure Data Lake Analytics<\/a><\/p>\n Companies collectively own Exabytes of data across hundreds of billions of streams, files, tables, reports, spreadsheets, VMs, etc. But for the most part, companies don't know where their data is, what's in it, how it's being used, or if it's secure. Our job is to fix that. We are building an automated infrastructure to find and classify any form of data anywhere it lives, from on-premises to cloud, figure out what the data contains, and how it should be managed\/protected. Our global scale infrastructure will track all of this data and provide a search engine to make the data discoverable and easily re-usable. From AI to distributed systems we are using cutting edge technologies to make data useful.<\/p>\n Create, schedule, and manage your data integration at scale. Work with data wherever it lives, in the cloud or on-premises, with enterprise-grade security. Accelerate your data integration projects by taking advantage of over 70 available data source connectors. Use the graphical user interface to build and manage your data pipelines. Transform raw data into finished, shaped data ready for consumption by business intelligence tools or custom applications. Easily lift your SQL Server Integration Services (SSIS) packages to Azure, and let Azure Data Factory manage your resources so you can increase productivity and lower total cost of ownership (TCO).<\/p>\n Azure Data Factory<\/a><\/p>\n Azure HDInsight is a fully-managed cluster service that makes it easy, fast, and cost-effective to process massive amounts of data. Use popular open-source frameworks such as Hadoop, Spark, Hive, LLAP, Kafka, Storm, ML Services, and more. Azure HDInsight enables a broad range of scenarios such as ETL, Data Warehousing, Machine Learning, IoT, and more. A cost-effective service that is powerful and reliable. Pay only for what you use. Create clusters on demand, then scale them up or down. Decoupled compute and storage provide better performance and flexibility.<\/p>\n Azure HDInsight<\/a><\/p>\n The world\u2019s first intelligent database service that learns and adapts with your application, enabling you to dynamically maximize performance with very little effort on your part. Working around the clock to learn, profile, and detect anomalous database activities, Threat Detection identifies potential threats to the database, and like an airplane\u2019s flight data recorder, Query Store collects detailed historical information about all queries, greatly simplifying performance forensics by reducing the time to diagnose and resolve issues.<\/p>\n Azure SQL Database<\/a><\/p>\n Adaptive query processing support for Azure SQL Database<\/a><\/p>\n Improved Automatic Tuning boosts your Azure SQL Database performance<\/a><\/p>\n Query Store: A flight data recorder for your database<\/a><\/p>\n A fast, fully managed, elastic, petabyte-scale cloud data warehouse. Azure SQL Data Warehouse (SQL DW) lets you independently scale compute and storage, while pausing and resuming your data warehouse within minutes through a distributed processing architecture designed for the cloud. Seamlessly create your hub for analytics along with native connectivity with data integration and visualization services, all while using your existing SQL and BI skills. Through PolyBase, Azure Data Warehouse brings data lake data into the warehouse with support for rich queries over files and unstructured data.<\/p>\n Azure SQL Data Warehouse<\/a><\/p>\n Blazing fast data warehousing with Azure SQL Data Warehouse<\/a><\/p>\n Azure SQL Database Threat Detection, your built-in security expert<\/a><\/p>\n Azure SQL Data Warehouse now generally available in all Azure regions worldwide<\/a><\/p>\n Easily develop and run massively parallel real-time analytics on multiple IoT or non-IoT streams of data using a simple SQL-like language. Use custom code for advanced scenarios. With no infrastructure to manage, you can process data on-demand, scale instantly, and only pay per job. Azure Stream Analytics seamlessly integrates with Azure IoT Hub and Azure IoT Suite. Azure Stream Analytics is also available on Azure IoT Edge enabling near-real-time analytical intelligence closer to IoT devices so that they can unlock the full value of device-generated data.<\/p>\n Azure Stream Analytics<\/a><\/p>\n The Azure Data Office of the CTO is the home of the Gray Systems Lab (GSL) and the Cloud and Information Services Lab (CISL). These are applied research groups, employing scientists and engineers focusing on everything data, from SQL Server in the cloud to massive-scale distributed systems for Big Data, to hardware-accelerated data processing to self-healing streaming solutions. The labs were founded in 2008 and 2012 respectively, have since produced 70+ papers in top database and system conferences, have filed numerous patents, and are continuously engaged in deep academic collaborations, through internships and sponsorship of academic projects. This research focus is matched with an equally strong commitment to open-source (with several committers in Apache projects who together contributed over 500K LOC to open-source), and continuous and deep engagement with product teams.<\/p>\n Strong product partnerships provide an invaluable path for innovation to affect the real world\u2014the labs\u2019 innovations today govern hundreds of thousands of servers in our Big Data infrastructure, and ship with several cloud and on-premises products. Microsoft\u2019s cloud-first strategy gives researchers in our group a unique vantage point, where PBs of telemetry data and first-party workloads make every step of our innovation process principled and data-driven. The labs have locations in Microsoft\u2019s Redmond and Silicon Valley campuses, as well as in the UW-Madison campus. To hear more, please contact Carlo Curino at carlo.curino@microsoft.com<\/a> or Alan Halverson at alanhal@microsoft.com<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" Hello VLDB attendees! Welcome to Rio De Janeiro. I wanted to take this opportunity to share with you some of the exciting work in data that\u2019s going on in the Azure Data team at Microsoft, and to invite you to take a closer look.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"ms_queue_id":[],"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","_alt_title":"","footnotes":"","msx_community_cta_settings":[]},"categories":[1473],"tags":[],"audience":[3057,3055,3056],"content-type":[1460],"product":[1539],"tech-community":[],"topic":[],"coauthors":[51],"class_list":["post-2369","post","type-post","status-publish","format-standard","hentry","category-databases","audience-data-professionals","audience-developers","audience-it-implementors","content-type-events","product-azure-sql-database"],"yoast_head":"\n
\n________________________________________<\/p>\nAzure Data blog posts<\/h2>\n
SQL Server<\/h2>\n
Azure Cosmos DB: The industry\u2019s first globally-distributed, multi-model database service<\/h2>\n
Azure Data Lake Analytics – An on-demand analytics job service to power intelligent action<\/h2>\n
Azure Data Catalog \u2013 Find, understand, and govern data<\/h2>\n
Azure Data Factory \u2013 Managed, hybrid data integration service at scale<\/h2>\n
Azure HDInsight \u2013 Managed cluster service for the full spectrum of Hadoop and Spark<\/h2>\n
Azure SQL Database – The intelligent relational cloud database service<\/h2>\n
Azure SQL Data Warehouse – Fast, flexible, and secure analytics platform<\/h2>\n
Azure Stream Analytics – A serverless, real-time analytics service to power intelligent action<\/h2>\n
Gray Systems Lab (GSL) and Cloud and Information Services Lab (CISL)<\/h2>\n