{"id":1737,"date":"2019-01-31T00:00:00","date_gmt":"2019-01-31T00:00:00","guid":{"rendered":"https:\/\/azure.microsoft.com\/blog\/transitioning-big-data-workloads-to-the-cloud-best-practices-from-unravel-data"},"modified":"2025-06-12T05:29:21","modified_gmt":"2025-06-12T12:29:21","slug":"transitioning-big-data-workloads-to-the-cloud-best-practices-from-unravel-data","status":"publish","type":"post","link":"https:\/\/azure.microsoft.com\/en-us\/blog\/transitioning-big-data-workloads-to-the-cloud-best-practices-from-unravel-data\/","title":{"rendered":"Transitioning big data workloads to the cloud: Best practices from Unravel Data"},"content":{"rendered":"\n

Migrating on-premises Apache Hadoop\u00ae and Spark workloads to the cloud remains a key priority for many organizations. In my last post, I shared \u201cTips and tricks for migrating on-premises Hadoop infrastructure to Azure HDInsight<\/a>.\u201d In this series, one of HDInsight\u2019s partners,\u202fUnravel Data<\/a>, will share their learnings, best practices, and guidance based on their insights from helping migrate many on-premises Hadoop and Spark deployments to the cloud.<\/p>\n\n\n\n

Unravel Data is an AI-driven Application Performance Management (APM) solution for managing and optimizing big data workloads. Unravel Data provides a unified, full-stack view of apps, resources, data, and users, enabling users to baseline and manage app performance and reliability, control costs and SLAs proactively, and apply automation to minimize support overhead. Ops and Dev teams use Unravel Data\u2019s unified capability for on-premises workloads and to plan, migrate, and operate workloads on Azure. Unravel Data is available on the\u202fHDInsight Application Platform.<\/p>\n\n\n\n

Today\u2019s post, which kicks off the five-part series, comes from Shivnath Babu, CTO and Co-Founder at Unravel Data. This blog series will discuss key considerations in planning for migrations. Upcoming posts will outline the best practices for the migration, operation, and optimization phases of the cloud adoption lifecycle for big data<\/a>.<\/p>\n\n\n\n

Unravel Data\u2019s perspective on migration planning<\/h2>\n\n\n\n

The cloud is helping to accelerate big data adoption across the enterprise. But while this provides the potential for much greater scalability, flexibility, optimization, and lower costs for big data, there are certain operational and visibility challenges that exist on-premises that don\u2019t disappear once you\u2019ve migrated workloads away from your data center.<\/p>\n\n\n\n

Time and time again, we have experienced situations where migration is oversimplified and considerations such as application dependencies and system version mapping are not given due attention. This results in cost overruns through over-provisioning or production delays through provisioning gaps.<\/p>\n\n\n\n

Businesses today are powered by modern data applications that rely on a multitude of platforms. These organizations desperately need a unified way to understand, plan, optimize, and automate the performance of their modern data apps and infrastructure. They need a solution that will allow them to quickly and intelligently resolve performance issues for any system through full-stack observability and AI-driven automation. Only then can these organizations keep up as the business landscape continues to evolve, and be certain that big data investments are delivering on their promises.<\/p>\n\n\n\n

Current challenges in big data<\/h2>\n\n\n\n

Today, IT uses many disparate technologies and siloed approaches to manage the various aspects of their modern data apps and big data infrastructure.<\/p>\n\n\n\n

Many existing monitoring solutions often do not provide end-to-end support for big data environments, lack full-stack compatibility, or require complex instrumentation. This includes configuration changes to applications and their components, which requires deep subject matter expertise. The murky soup of monitoring solutions that organizations currently rely on doesn\u2019t deliver the application agility that is required by the business.<\/p>\n\n\n\n

Consequently, this results in poor user experience, inefficiencies and mounting costs as organizations buy more and more tools to solve these problems and then have to spend additional resources managing and maintaining those tools.<\/p>\n\n\n\n

Additionally, organizations see a high Mean Time to Identify (MTTI) and Mean Time to Resolve (MTTR) issues because it is hard to understand the dependencies and keep focused on root cause analysis. The lack of granularity and end to end visibility makes it impossible to remedy all of these problems, and businesses are stuck in a state of limbo.<\/p>\n\n\n\n

It\u2019s not an option to continue doing what was done in the past. Teams need a detailed appreciation of what they are doing today, what gaps they still have, and what steps they can take to improve business outcomes. It\u2019s not uncommon to see 10x or more improvements in root cause analysis and remediation times for customers who are able to gain a deep understanding of the current state of their big data strategy and make a plan for where they need to be.<\/p>\n\n\n\n

Starting your big data journey to the cloud<\/h2>\n\n\n\n

Without a unified APM platform, the challenges only intensify as enterprises move big data to the cloud. Cloud adoption is not a finite process with a clear start and end date \u2014 it\u2019s an ongoing lifecycle with four broad phases (planning, migration, operation, and optimization). Below, we briefly discuss some of the key challenges and questions that arise for organizations below, which we will dive into in further detail in subsequent posts.<\/p>\n\n\n\n

In the\u202fplanning\u202f<\/strong>phase, key questions may include:<\/p>\n\n\n\n