Microsoft Azure Blog

Use New Relic One to effortlessly monitor applications in Azure Spring Cloud

Asir Selvasingh — Thu, 01 Jul 2021 08:00:27 Z

Today, we are announcing the integration of New Relic One performance monitoring in Azure Spring Cloud.

Over the past 18 months, we worked with many enterprise customers to learn about their scenarios. Many of these customers have thousands of Spring Boot applications running in on-premises data centers. As they migrate these applications to the cloud, they need to instrument them for application performance monitoring (APM) using tools that their developers are familiar with, and have been using for years, and they must ensure continuity for desktop and mobile applications which are already pre-instrumented for end-to-end monitoring using agents like New Relic One. With the integration of New Relic One in Azure Spring Cloud, you can continue your journey and easily instrument your Spring Boot applications with New Relic One.

Most organizations that deploy Spring Boot applications today share a similar goal: maximize the benefits of running Spring Boot applications at virtually any scale, using automation and application performance monitoring. While Azure Spring Cloud excels at abstracting away much of the toil associated with managing containerized workloads, the challenge of monitoring and maintaining the performance and health of these applications, or of troubleshooting issues when they occur, can be daunting—especially as organizations deploy these applications at massive scale. To help you succeed and continue your New Relic One journey, we integrated and upgraded your ability to instrument, monitor, and deliver observability using New Relic One across your Azure Spring Cloud instances. That begins with setting up instrumentation quickly and easily. Then you can analyze the performance and health of your applications, Java Virtual Machines (JVM), transactions, and more to identify and troubleshoot performance issues.

Instrument your Spring Boot applications

Run a "provisioning" automation for a completely hands-off experience for instrumenting and monitoring any new applications that you create and deploy—using Terraform or ARM Template. Or you can run it on-demand using the Azure CLI for greater flexibility and control.


az spring-cloud app update --name customers-service \
    --jvm-options="-javaagent:/opt/agents/newrelic/java/newrelic-agent.jar" \
    --env NEW_RELIC_APP_NAME=customers-service NEW_RELIC_LICENSE_KEY=<key>

Explore Spring Boot application performance

After instrumenting, you can explore Spring Boot application performance through the New Relic One portal. You can view your applications in action through the Service Maps. For example, the screenshot below shows all the applications in the popular Petclinic project:

Figure 1: shows all the applications in the popular Petclinic project.

You can observe your application’s performance by navigating through application-by-application in the New Relic One portal. For example, the screenshot below shows transactions processed by customers-service, its throughput, and error rate:

Figure 2: shows transactions processed by customers-service, its throughput, and error rate.

You can look deeper into the underlying Java Virtual Machines by navigating to the “JVMs” blade on the New Relic One portal. For example, the screenshot below shows how many Java Virtual Machines for all the customers-service application instances:

Figure 3: shows how many Java Virtual Machines for all the customers-service application instances.

You can analyze application dependencies, including database transactions by navigating to the “Databases” blade in the New Relic One portal. For example, the screenshot below shows all the database transactions for customers-service:

Figure 4: shows all the database transactions for customers-service.

You can look at the stack trace of each thread through the New Relic One Thread Profiler and understand times consumed by Java methods. This provides insights into the "hot" functions of the applications where most time is spent. For example, the screenshot below shows a stack trace of a thread in the customers-service application:

Figure 5: shows a stack trace of a thread in the customers-service.

Deliver value to your end-users

Once instrumented, as you scale out to multiple Spring Boot application instances, any new application instances are automatically monitored for you. You spend less time managing the agent installation and maintenance, and more energy on identifying and resolving incidents faster. Azure Spring Cloud service is on-point for periodically updating the agent.

View of your Spring Boot application instances

New Relic One Navigator offers you an at-a-glance view of the operational health of your application instances. This bird's eye view of your estate is automatically available to you, no configuration required. Navigator makes it easy for you to explore huge numbers of instances as it intuitively displays your instances in a dense honeycomb view with traffic light colors based on alerts that you define. Quickly identify alerting instances and uncover root causes in addition to seeing which dependent systems might be affected.

As an illustration, the following Navigator view groups your instances across Azure Spring Cloud instances, allowing you to quickly compare your operations in a multi-region deployment. You can group and filter across your instances to quickly zero in on issues. You can also drill down into any instance to see a summary of its activity and any violations along with key metrics and metadata including custom attributes:

Figure 6: show how Navigator view groups your instances across Azure Spring Cloud instances.

Detect anomalies in your Spring Boot application instances

New Relic Lookout provides an intuitive view of your instances that are deviating from normal behavior, using circle visualization with color indicating the severity of recent signal change and sized proportionally to the magnitude of the metric signal. Lookout automatically compares signals within the last five minutes against the previous hour.

Similarly, you can use New Relic One for monitoring Java applications in Azure services such as Azure Kubernetes Service and Virtual Machines.

Build your solutions and monitor them today

Azure Spring Cloud is jointly built, operated, and supported by Microsoft and VMware. It is a fully managed service for Spring Boot applications that abstracts away the complexity of infrastructure and Spring Cloud middleware management, so you can focus on building your business logic and let Azure take care of dynamic scaling, patches, security, compliance, and high availability. With a few steps, you can provision Azure Spring Cloud, create applications, deploy, and scale Spring Boot applications, and start monitoring in minutes. We will continue to bring more developer-friendly and enterprise-ready features to Azure Spring Cloud.

We would love to hear how you are building impactful solutions using Azure Spring Cloud. Get started today—deploy Spring applications to Azure Spring Cloud using quickstart.

Resources

Read the companion announcement blog from New Relic.
Learn using a Microsoft Learn module or self-paced workshop on GitHub.
Learn more about implementing solutions on Azure Spring Cloud.
Learn more about New Relic Java agent in Azure Spring Cloud.
Deploy a distributed version of Spring Petclinic built with Spring Cloud.
Deploy Spring Boot applications by leveraging enterprise best practices—Azure Spring Cloud reference architecture.
Migrate your Spring Boot, Spring Cloud, and Tomcat applications to Azure Spring Cloud.
Wire Spring applications to interact with Azure services.
For feedback and questions, please email us.

Advancing safe deployment with AIOps—introducing Gandalf

Mark Russinovich — Wed, 30 Jun 2021 09:00:13 Z

“Changes to Azure services and the Azure platform itself are both inevitable and beneficial, to ensure continuous delivery of updates, new features, and security enhancements. However, change is also a primary cause of service regressions that can contribute towards reliability issues—for hyperscale cloud providers, indeed for any IT service provider. As such, it is critical to catch any such problems as early as possible during the development and deployment rollout, to minimize any impact on the customer experience. As part of our ongoing Advancing Reliability blog series, today I’ve asked Principal Program Manager Jian Zhang from our AIOps team to introduce how we’re increasingly leveraging machine learning to de-risk these changes, ultimately to improve the reliability of Azure.”—Mark Russinovich, CTO, Azure

This post includes contributions from Principal Data Scientists Ken Hsieh and Ze Li, Principal Data Scientist Manager Yingnong Dang, and Partner Group Software Engineering Manager Murali Chintalapati.

In our earlier blog post “Advancing safe deployment practices” Cristina del Amo Casado described how we release changes to production, for both code and configuration changes, across the Azure platform. The processes consist of delivering changes progressively, with phases that incorporate enough bake time to allow detection at a small scale for most regressions missed during testing.

The continuous monitoring of health metrics is a fundamental part of this process, and this is where AIOps plays a critical role—it allows the detection of anomalies to trigger alerts and the automation of correcting actions such as stopping the deployment or initiating rollbacks.

In the post that follows, we introduce how AI and machine learning are used to empower DevOps engineers, monitor the Azure deployment process at scale, detect issues early, and make rollout or rollback decisions based on impact scope and severity.

Why AIOps for safe deployment

As defined by Gartner, AIOps enhances IT operations through insights that combine big data, machine learning, and visualization to automate IT operations processes, including event correlation, anomaly detection, and causality determination. In our earlier post, "Advancing Azure service quality with artificial intelligence: AIOps," we shared our vision and some of the ways in which we are already using AIOps in practice, including around safe deployment. AIOps is well suited to catching failures during deployment rollout, particularly because of the complexities of cross-service dependencies, the scale of hyperscale cloud services, and the variety of different customer scenarios supported.

Phased rollouts and enriched health signals are used to facilitate monitoring and decision making in the deployment process, but the volume of signals and level of complexity involved in deployment decision making exceeds what any human could reasonably reason over, across thousands of ever-evolving service components, spanning more than 200 datacenters in more than 60 regions. Some latent issues won’t manifest for several days after their deployment, and global issues that span different clusters but manifest only minutely in any individual cluster are hard to detect with just a local watchdog. While loose coupling allows most service components to be deployed independently, their deployments could have intricate impacts on each other. For example, a simple change in an upstream service could potentially impact a downstream service if it breaks the contract of API calls between the two services.

These challenges call for automated monitoring, anomaly detection, and rollout impact assessment solutions to facilitate deployment decisions at velocity.

Figure 1: Gandalf safe deployment

Gandalf safe deployment service: An AIOps solution

Rising to the challenge described above, the Azure Compute Insights team developed the “Gandalf” safe deployment service—an end-to-end, continuous monitoring system for safe deployment. We consider this part of the Gandalf AIOps solution suite, which includes a few other intelligent monitoring services. The code name Gandalf was inspired by the protagonist from The Lord of the Rings, as shown in Figure 1, it serves as a global watchdog, which makes intelligent deployment decisions based on signals collected. It works in tandem with local watchdogs, safe deployment policies, and pre-qualification tests, all to ensure deployment safety and velocity.

As illustrated in Figure 2, the Gandalf system monitors rich and representative signals from Azure, performs anomaly detection and correlation, then derives insights to support deployment decision making and actions.

Figure 2: Gandalf system overview

Data sources

Gandalf monitors signals across performance, failures, and events as described below. It pre-processes the data to structure them around a unified data schema to support downstream data analytics. It also leverages a few other analytics services within Azure for health signals, including our Virtual Machine failure categorization service and near real-time failure attribution processing service. Signal registration with Gandalf is required when any new service components are onboarded, to ensure complete coverage.

Performance data: Gandalf monitors performance counters, CPU usage, memory usage, and more – all for a high-level view of performance and resource consumption patterns of hosted services.
Failure signals: Gandalf monitors both the hosting environment of customer’s virtual machines (data plane) and tenant-level services (control plane). For the data plane, it monitors failure signals such as OS crashes, node faults, and reboots to evaluate the health of the VM’s hosting environment. At the same time, it monitors failure signals of the control plane like API call failures, to evaluate the health of tenant-level services.
Update events: In addition to telemetry data collected, Gandalf also keeps its finger on the pulse of deployment events, which report deployment progress and issues.

Detection, correlation, and decision

Gandalf evaluates the impact scope of the deployment—for example, the number of impacted nodes, clusters, and customers—to make a go/no-go decision using decision criteria that are trained dynamically. To balance speed and coverage, Gandalf utilizes an architecture with both streaming and batch analysis engines.

Figure 3: Gandalf Anomaly Detection and Correlation Mode

Figure 3 shows an overview of the Gandalf Machine Learning (ML) model. It consists of two parts—anomaly detection and correlation process (to identify suspicious deployments) and a decision process (to evaluate customer impact).

Anomaly detection and correlation process

To ensure precise detection, Gandalf derives fault signatures from input signals, which can be used to uniquely identify the failure. Then, it detects based on the occurrence of the fault signature.

In large-scale cloud systems like Azure, simple threshold-based detection is not practical both because of the dynamic nature of the systems and workloads hosted and because of the sheer volume of fault signatures. Gandalf applies machine learning techniques to estimate baseline settings based on historical data automatically and can adapt the setting through training as needed.

When Gandalf detects an anomaly, it correlates the observed failure with deployment events and evaluates its impact scope. This helps to filter out failures caused by non-deployment reasons such as random firmware issues.

Since multiple system components are often deployed concurrently, a vote-veto mechanism is used to establish the relationship between the faults and the rollout components. In addition, temporal and spatial correlations are used to identify the components at fault. Fault age, which measures the time between rollout and detection of fault signature, is considered to allow more focus on new rollouts than old ones since newly observed faults are less likely to be triggered by the old rollout.

In this way, Gandalf can detect an anomaly that would lead to potential regressions in the customer experience early in the process—before it generates widespread customer impact. For more detail, refer to our published paper “Gandalf: An Intelligent, end-to-end analytics service for safe deployment in large-scale cloud infrastructure.”

Decision process

Finally, Gandalf evaluates the impact scope of the deployment such as the number of impacted clusters/nodes/customers, and ultimately makes a "go/no-go" decision. It’s worth mentioning that Gandalf is designed to allow developers to customize signals’ weight assignment based on their experience. In this way, it can incorporate domain knowledge from human experts to complement its machine learning solutions.

Result orchestration

To balance speed and coverage, Gandalf utilizes both streaming and batch processing of incoming signals. Streaming processing consumes data from Azure Data Explorer, a cloud storage solution supporting analytics with fast speed. Streaming processing is used to process fault signals that happen 1 hour before and after each deployment in each node and runs lightweight analysis algorithms for rapid response.

Batch processing consumes data from Cosmos, a Hadoop-like file system that supports extremely large volumes of data. It’s used to analyze faults over a larger time window (generally a 30-day period) with advanced algorithms.

Both stream and batch processing are performed incrementally with five-minute intervals. In general, the incoming telemetry signals of Gandalf are both streamed into Kusto and stored into Cosmos hourly/daily. With the same data source, occasionally there could be inconsistent results from the processing pipeline. This is by design since batch processing makes more informed decisions and covers latent issues that the fast/streaming process cannot detect.

Deployment experience transformation

The Gandalf system is now well integrated into our DevOps workflow within Azure and has been widely adopted for deployment health monitoring across the entire fleet. It not only helps to prevent bad rollouts as quickly as possible but has also transformed the engineers’ and release managers’ experience in deploying software changes—from looking for scattered evidence to using a single source of truth, from ad-hoc diagnoses to using interactive troubleshooting—and in so doing, many of the engineers who interact with Gandalf have had their opinions on it transformed as well, evolving from skeptics to advocates.

In many Azure services, Gandalf has become a default baseline for all release validations, and it’s exciting to hear how much our on-call engineers trust Gandalf.

Summary

In this post, we have introduced the Gandalf safe deployment service, an intelligent, end-to-end analytics service for the safe deployment of Azure services. Through state-of-the-art anomaly detection, special and temporal correlation, and result orchestration, the Gandalf safe deployment service enables DevOps engineers to make go/no-go decisions accurately, and with the velocity needed by hyper-scale cloud platforms like Azure.

We will continue to invest in applying AI- and machine learning-based technologies to improve cloud service management, ultimately to continue improving the customer experience. Look for us to share more about our AIOps solutions, including pre-production analytics to further help us push quality to the left.

Azure Cost Management and Billing updates – June 2021

Michael Flanakin — Tue, 29 Jun 2021 10:00:07 Z

Whether you’re a new student, a thriving startup, or the largest enterprise, you have financial constraints, and you need to know what you’re spending, where, and how to plan for the future. Nobody wants a surprise when it comes to the bill, and this is where Azure Cost Management and Billing comes in.

We’re always looking for ways to learn more about your challenges and how Azure Cost Management and Billing can help you better understand where you’re accruing costs in the cloud, identify and prevent bad spending patterns, and optimize costs to empower you to do more with less. Here are a few of the latest improvements and updates based on your feedback:

Viewing amortized costs in the cost analysis preview.
Reminder: Cloudyn retiring on June 30.
What's new in Cost Management Labs.
New ways to save money with Azure.
Documentation updates.

Let’s dig into the details.

Viewing amortized costs in the cost analysis preview

The cost analysis preview introduced a new Reservations view when it was first announced in November 2020. The Reservations view shows the total amortized cost for each reservation with a breakdown by resource to facilitate chargeback. Now that same option has been extended to other preview views, allowing you to view amortized cost for your resources, resource groups, or subscriptions. Simply open the view you need, then select Customize and switch from Actual cost to Amortized cost.

If you're not familiar, amortization splits the purchase price of your reservations into daily chunks and allocates these out to the resources that received the benefit. Let's say you purchased a VM reservation for $365 on January 1. Enabling amortization will show $1 every day throughout the year. If your VM ran the full day, it'll get the full $1 for that day. If you only ran it for 12 hours one day, then you'll see $0.50 for the VM and $0.50 for the unused reservation. This is lost savings from your pre-paid, reserved capacity, so you'll want to keep an eye on that. This is exactly what the Reservations view helps you do. And beyond that, what you can use amortized cost for.

Keep using the preview and let us know what you'd like to see next.

Reminder: Cloudyn retiring on June 30

Microsoft acquired Cloudyn in July 2017 to bring a wealth of cost management experience and capabilities to the broader Azure community. Many organizations have known and loved the capabilities Cloudyn brought to the table and now that many of these have been (or are being) integrated into Azure Cost Management, Cloudyn will be retiring on June 30, 2021. Whether you relied on cross-cloud cost governance or cost allocation and chargeback, or maybe you simply need basic cost reporting, accountability, and optimization, Azure Cost Management has you covered. And that's not all. There's even more in the roadmap coming this year, so stay tuned.

If you're a Cloud Solution Provider still using Cloudyn, we've already reached out to you, but as a reminder, you (and your customers) will be able to view, manage, and optimize costs using Azure Cost Management once you transition to Microsoft Customer Agreement and an Azure plan. Again, Cloudyn will be retiring on June 30, 2021, and will no longer be accessible. We thank you for your patronage and look forward to supporting you and your financial goals with Azure Cost Management.

What's new in Cost Management Labs

With Cost Management Labs, you get a sneak peek at what’s coming in Azure Cost Management and can engage directly with us to share feedback and help us better understand how you use the service, so we can deliver more tuned and optimized experiences. Here are a few features you can see in Cost Management Labs:

Remember last-used scope – Now available in the public portal
Azure Cost Management now remembers the last scope you used, making it easier than ever to pick up where you left off. You can see the preview in Cost Management Labs.
New: Confirm billing policy status in Configuration
Updated Configuration screen that separates scope and cost management settings and shows whether Cost Management, Reservations, and Azure Marketplace are enabled for your Enterprise Agreement billing account or Microsoft Customer Agreement billing profile.
View cost for your resources
The cost for your resources is one click away from the resource overview in the preview portal. Just click View cost to quickly jump to the cost of that particular resource.
Change scope from the menu
Change scope from the menu for quicker navigation. You can opt-in using Try Preview. Share feedback about this preview.
Open configuration items in the main menu
This is an experimental option to show the selected configuration screen as a nested menu item in the Cost Management menu. You can opt-in using Try Preview from Cost Management Labs. Share feedback about this preview.
Streamlined Cost Management menu
Only show settings in Configuration. Remove Cloudyn, Exports, and Connectors for AWS from the Cost Management menu. You can opt-in using Try Preview from Cost Management Labs. Share feedback about this preview.

Of course, that’s not all. Every change in Azure Cost Management is available in Cost Management Labs a week before it’s in the full Azure portal. We’re eager to hear your thoughts and understand what you’d like to see next. What are you waiting for? Try Cost Management Labs today.

New ways to save money with Azure

This month was packed with 16 new and upcoming ways to help you optimize your costs:

General availability: App Service managed certificates.
General availability: Cosmos DB expanded free tier.
General availability: Cosmos DB Serverless.
General availability: 5 new regions for Azure Stream Analytics – Australia Central, Germany West Central, Norway East, South India, UAE North.
General availability: 8 new regions for Azure Health Bot – Australia East, East US 2, India Central, North Europe, South Central US, Southeast Asia, UK South, West US 2.
Lower price: Azure Sentinel and Azure Monitor.
Lower price: ExpressRoute Global Reach.
Lower price: API Management Premium.
Free trial extended: Azure Stack hyperconverged infrastructure (HCI).
Preview: Azure SQL Database ledger.
Preview: Cosmos DB cost-optimized diagnostics.
Preview: Cosmos DB Linux emulator.
Preview: Cosmos DB integrated cache.
Preview: Event Hubs Premium.
Coming soon: Azure Database for PostgreSQL free offer.
Coming soon: Azure Database for MySQL free offer.

Documentation updates

Here are a few documentation updates you might be interested in:

Want to keep an eye on all of the documentation updates? Check out the Cost Management and Billing doc change history in the azure-docs repository on GitHub. If you see something missing, select Edit at the top of the document and submit a quick pull request.

What’s next?

These are just a few of the big updates from last month. Don’t forget to check out the previous Azure Cost Management and Billing updates. We’re always listening and making constant improvements based on your feedback, so please keep the feedback coming.

Follow @AzureCostMgmt on Twitter and subscribe to the YouTube channel for updates, tips, and tricks. You can also share ideas and vote up others in the Cost Management feedback forum or join the research panel to participate in a future study and help shape the future of Azure Cost Management and Billing.

We know these are trying times for everyone. Best wishes from the Azure Cost Management and Billing team. Stay safe and stay healthy.

Bring cloud experiences to data workloads anywhere with Azure SQL enabled by Azure Arc

Rohan Kumar — Tue, 29 Jun 2021 07:00:05 Z

From edge to cloud, companies are eager to find innovative solutions that meet them where they are. Today’s business environment is increasingly complex, and customers tell us they need solutions that are multi-cloud, platform-agnostic, and offer integrated apps and services that are always up to date. We’ve seen firsthand the best solution combines a company’s existing infrastructure with the simple, distributed computing power of the cloud. Our unique approach to hybrid gives businesses choice and flexibility in managing their entire data estate while complying with data sovereignty, regulation, and latency requirements.

Tune in to the Azure Hybrid and Multi-cloud Digital Event from 9:00 – 11:00 AM Pacific Time to learn more about the latest announcements about Azure Arc-enabled data services. And today, we’re announcing the general availability of Azure SQL enabled by Azure Arc to help customers manage their data and databases simply and efficiently. Customers can deploy cloud services on-premises and in multi-cloud environments—effectively modernizing in place by running Azure SQL on any infrastructure.

Cloud solutions that simplify the most complex hybrid data scenarios

Deploying cloud-based solutions to on-premises and multi-cloud environments helps deliver consistent, portable, and unified management across a company’s entire data estate. Our customers are seeing improved productivity and operating efficiency by using hybrid technology to manage their data and app development at scale.

For example, SKF provides reliable rotation to industries all over the world, offering products and services around the rotating shaft including bearings, seals, lubrication management, artificial intelligence, and wireless condition monitoring. SKF has been on a journey to digitally transform the company’s backbone through harnessing the power of technology, interconnecting processes, streamlining operations, and delivering industry-leading digital products and services for customers. SKF uses Azure Arc, along with Azure Stack HCI and Azure SQL Edge, to scale solutions and standardize processes across their 91 manufacturing sites, resulting in 40 percent savings on hardware costs and 30 percent savings in OT-related machine downtime.

“SKF’s focus is on digitalizing all segments of the value chain and interconnecting them to unlock the full potential of digital ways of working for our business and customers. Azure Arc-enabled SQL Managed Instance is providing us worry-free and always up-to-date SQL operations. These types of services provide us a tremendous jumpstart into digitalization.”—Sven Vollbehr, Head of Digital Manufacturing, SKF

The “anywhere” solution for hybrid cloud applications

Beginning July 30, 2021, the general-purpose tier of Azure Arc-enabled SQL Managed Instance will be generally available, providing database-as-a-service (DBaaS) functionality on any infrastructure. Businesses will be able to deploy Azure SQL databases on any infrastructure and on any Kubernetes to:

Stay current with automated updates and deploy evergreen feature and security updates to on-prem databases with no end-of-support.
Automate routine database administrator (DBA) tasks at scale with built-in management capabilities including high availability, backup, and restore.
Optimize data workload performance by bringing cloud elasticity on-premises for existing infrastructure, using only the resources needed to dynamically scale up, down, without application downtime.
Access Azure industry-leading security and governance capabilities for your on-premises data workloads using to protect your data.

New product features and capabilities will be rolled out on a continuous basis, and customers will be able to opt-in to preview additional Azure Arc-enabled services, such as PostgreSQL, and easily integrate as they become generally available. Additionally, customers will have the opportunity to test out future road map preview features and provide early feedback.

Develop breakthrough applications with our network of trusted partners

In support of Azure Arc-enabled data services' general availability, we are announcing an expanded network of trusted partners and validated solutions to help customers get started.

Our team works closely with technology providers to validate popular platforms to work with Azure Arc, and our many service providers are here to provide customers with the latest innovations for hybrid data solutions on-premises or in multi-cloud environments.

Whether you are just getting started with migration and modernization efforts or in the middle of a multi-year digital transformation, our consulting services partners can help you choose the validated infrastructures and applications that are specifically configured and tested to work with Azure Arc.

Get started with Azure Arc-enabled data services

Get started with Azure Arc-enabled data services available in preview today. Starting July 30, 2021, Azure Arc-enabled SQL Managed Instance (general-purpose tier) will be generally available for customers to optimize their data workloads from cloud to edge.

Tune in to the Azure Hybrid and Multi-cloud Digital Event today at 9:00 AM Pacific Time or watch on-demand to learn more about the announcement.

Get started and view frequently asked questions.
Review Azure Arc documentation and Jumpstart scenarios.

Streamline your custom image building process with Azure VM Image Builder service

Michael Yen-Chi Ho — Thu, 24 Jun 2021 09:00:02 Z

Today we are announcing the general availability of the Azure VM Image Builder service, a managed service for you to build custom Linux or Windows virtual machine (VM) images with ease, and be compliant with your company’s security policy across Azure and Azure Stack.

With Azure VM Image Builder, the Microsoft managed service built on HashiCorp Packer, you can describe custom images in a template using new or existing configurations and enables VM image building immediately without setting up and managing your own image building pipeline.

Image building has always been an important yet complex process; you start with collecting the requirements from different teams within your organization, and then you need to start building the image configuration tooling and set up the right infrastructure to support the implementation of making a new custom golden image. Finally, if you were to build many images at once, an automation configuration is needed so you can get everything in the funnel and start working.

During the process, you also need to have a group of experts covering each step to ensure the process runs smoothly. The process gets even more complicated if you need to build images for different Operating Systems and deploy them on multiple clouds. This is because every Operating System comes with different configuration styles and patterns and every cloud has different image configuration requirements.

Azure VM Image Builder allows you to describe a custom image for Azure using a single template. You don't need to relearn or rebuild all the tooling and image building pipelines; all you need to do is to gather your organization’s requirements and then choose from the template in Azure for building Linux or Windows VM images and the service will create the image for you. You don't need to worry about all the infrastructure, technical differences between Operating System versions, or other underlying components in the image building while getting what you need in a simple, streamlined way.

With Azure VM Image Builder, we’re making image building extremely easy for you with different options like using the Azure CLI or the Azure portal to start your VM Image Build process. And you can also combine the service with Azure Shared Image Gallery to set up your image-building pipeline and distribute the images globally with just a few clicks. Applying security configurations and group requirements will no longer be a heavy burden when it comes to building custom golden images.

Azure VM Image Builder service support scenarios

Streamline your Linux, Windows, or baseline image creation process that includes your minimum security and corporate configurations, and allows departments to customize it further.
Integrate core applications and configurations, so your VMs can take on workloads after the image creation process such as Windows Virtual Desktop configurations.
Patch and update your existing Linux or Windows images or create images from Azure Marketplace images.
Connect Azure VM Image Builder to your existing virtual networks, so you can connect your existing configuration servers (DSC, Chef, Puppet, etc.), file shares, or any other routable servers or services for quicker image building pipeline deployment on Azure.
Distribute images and scale globally with the Azure Shared Image Gallery, and apply your images across your deployments worldwide.
Integrate with your existing image building pipelines, just call Azure VM Image Builder from your pipeline, or use the simple Image Builder Azure DevOps Task (preview) to seamlessly connect and fit in your operations.

Get started with Azure VM Image Builder Service with ready to use examples

Manage your digital transformation at scale for retail with Cloud Adoption Framework for Azure

Jaishree Subramania — Wed, 23 Jun 2021 10:00:29 Z

Retail organizations are embracing digital transformation and cloud technologies to be more agile and achieve strategic goals. The cloud enables them to drive innovation, make more informed decisions with data analytics, outpace competitors, and deliver products, services, and applications faster. The path to the cloud is unique for every organization and navigating that journey can be a challenge.

Accelerating business transformation with Cloud Adoption Framework

We’re excited to share new guidance within the Cloud Adoption Framework for the retail industry. It amplifies the documentation, best practices, tools, templates, and other useful resources already included in the framework, to accelerate retail customers to achieve their strategic goals and transform their business. This new content provides guidance to help retail organizations navigate their cloud journey right from defining their business strategy, creating their adoption plan, preparing their cloud environments, implementing governance and operations to maximize business value.

Ideally, organizations start with building their cloud strategy by understanding their business motivations and the industry-specific opportunities they need to address, including:

Personalization: Consolidate customer data sources to drive relevant offers, grow revenue, and improve customer experiences.
Omnichannel optimization: Provide a seamless customer journey, enabling customers to switch platforms, devices, and channels, while still having a connected retail experience.
Supply chain optimization: Optimize supply chain with advanced insights and analytics that enhance product and service delivery and optimize planning to improve fulfillment, sourcing, and logistics.

Once the motivations and business outcomes have been identified, retail organizations would need to understand their digital maturity and assess where they are at in their cloud journey, to then properly plan how they would deliver on the business expectations and transform their business. To assist with this, we have created a retail maturity model that maps aspirational retail goals with recommended cloud technologies to potentially unlock those goals. By mapping the organization strategy, current and future stage, against this model, retail organizations would be able to prepare and execute a successful cloud adoption plan aligned to their business expectations.

Siloed retail

Little to no cloud adoption and underutilized data. Legacy patterns and practices need to be modernized.

Connected retail

Starting with migrating and modernizing technology solutions with the cloud to enable advanced innovations and reduce operational overhead.

Analytics-driven retail

Taking steps to align to a common data model and preparing for future growth, unlocking advanced analytics while improving data governance.

Intelligent retail

Integrating AI and machine learning into custom applications and Microsoft's retail cloud offerings to deliver intelligent retail and drive more value for customers.

After the business aspects of defining the strategy and creating the plan are in place, retail organizations can use the technical implementation guidance to build the proper cloud environment with Azure landing zones, and governance and management best practices to ensure that their cloud solutions are operating in alignment with their own and industry-specific security, compliance, and governance requirements.

Lastly, the scenario content also includes reference architectures to assist with the rapid deployment of retail solutions for common use cases, including scalable personalization on Azure, data warehousing and analytics, e-commerce front end, intelligent product search engine for e-commerce, and several others. You can find the full list of reference architectures and other solution areas for retail.

Real-world examples of how retail organizations address key challenges with Azure

Walgreens is able to rapidly analyze and generate insights from the tons of data it manages using Azure. With the power of Microsoft Azure Synapse Analytics, it has been able to dramatically improve certain operations, lower TCO, and modernize business processes and decision-making. Read how Walgreens modernized retail operations.

Albertsons migrated its entire datacenter to Microsoft Azure to leverage its capabilities like scalability, flexibility, resiliency, and agility to deliver a flexible omnichannel platform to customers that supports growth and innovation. Read how Albertsons accelerated cloud adoption by migrating its on-premises datacenter.

Learn more about how Azure drives digital transformation for retail organizations.

Get Started

Proper and thoughtful cloud adoption is key for retail organizations that want to reimagine the way they do business by:

Enhancing day-to-day operations with real-time data insights
Increasing customer engagement with personalized shopping experiences
Building resilient supply chains with end-to-end analytics and AI

The retail scenario in the Cloud Adoption Framework provides essential guidance, technical documentation, and resources to help retail organizations achieve digital transformation and get on the path to successful cloud use.

You can start today by reviewing the documentation and learning more about the Cloud Adoption Framework for Azure.

Join Microsoft at ISC 2021 and drive innovations with Azure high-performance computing

Kevin Raines — Wed, 23 Jun 2021 09:00:29 Z

Customers around the world rely on Microsoft Azure to drive innovations.

Whether it’s understanding our environment, public health, energy sustainability, weather modeling, economic growth, and many others finding solutions to these important challenges requires huge amounts of focused computing power. Customers are increasingly finding the best way to access such high-performance computing (HPC) is through the agility, scale, security, and leading-edge performance of Azure’s purpose-built HPC and AI cloud services.

Azure’s market-leading vision for HPC and AI is based on a core of genuine and recognized HPC expertise, using proven HPC technology and design principles, enhanced with the best features of the cloud. The result is a complete platform that delivers performance, scalability, and value, unlike any other cloud. This means applications scaling 12 times higher than other public clouds. It means higher application performance per VM. It means powering AI workloads for one customer with a supercomputer fit to be among the top five in the world. And it means delivering massive computing power into the hands of medical researchers over a weekend to prove out life-saving innovations in the fight against COVID-19.

Join us at ISC 2021 to learn more about Azure HPC and AI Supercomputing and Quantum

Microsoft will deliver a keynote at the International Supercomputing Conference (ISC) 2021 on quantum computing, and participate in multiple places across the ISC 2021 technical program, including panel sessions, talks, and presentations by our customers.

Additionally, we invite you to join us at an informal off-hour event at ISC on Monday, June 28, 2021, from 6:30 PM – 7:30 PM CEST for our inaugural “open community social.” We will have a range of HPC and Quantum experts from research, engineering, and customer-facing roles. We will also officially launch our Azure HPC community program. Learn additional details about the Azure HPC after-hour social event at ISC 2021 and join our Discord group prior to the off-hour event.

Check out our blog post featuring additional details about our ISC events and activities.

Announcements for Azure HPC and AI Supercomputing in 2021

Azure FX-series Virtual Machine general availability announcement

We are announcing the general availability of Azure FX-series Virtual Machine, specifically designed to optimize electronic design automation (EDA) workload processing with a purpose-built virtual machine architected for EDA workloads. For more details see the FX-series announcement Azure blog, "Optimize extreme computing performance with Azure FX-series Virtual Machines" as well as the Ansys RedHawk announcement, "How Azure FX VM Makes Ansys RedHawk-SC™ Run Faster the Less You Spend."

NVIDIA ND A100 v4 launch

Azure announced the general availability of scale-out NVIDIA A100 GPU Clusters with HDR 200 Gb/s InfiniBand networking: the fastest public cloud supercomputer. The ND A100 v4 VM is powered by NVIDIA A100 Tensor Core GPUs and is designed to let our most demanding customers scale up and scale out without ever slowing down.

Xilinx NP launch

Azure launched general availability of the Azure NP-Series Virtual Machine. The NP-series virtual machines are powered by Xilinx U250 FPGAs for accelerating workloads including machine learning inference, video transcoding, and database search and analytics. NP-series VMs are also powered by Intel Xeon 8171M (Skylake) CPUs with an all-core turbo clock speed of 3.2 GHz.

AMD HBv3 launch

Azure announced the general availability of the Azure HBv3 virtual machine for HPC workloads advancing the velocity at which we bring the latest technologies to our HPC customers and the compute performance we put at their fingertips.

Azure HPC and AI Collaboration Centers launch

Microsoft announced the HPC and AI Collaboration Centers program with five inaugural partners. These partners will share best practices for unlocking customer innovation and productivity with HPC and AI.

Big moments for Azure HPC and AI Supercomputing in 2021

UK Met Office supercomputing announcement

The UK Met Office supercomputing announcement highlighted that Microsoft had been awarded a contract to deliver a ten-year managed supercomputing service, providing advanced supercomputing capabilities for weather and climate research, ensuring the continuation of the Met Office’s leadership in this area.

Jellyfish

Jellyfish Pictures needed to enable secure remote access to immense computing power to render visual effects and animation sequences. The studio gained burst rendering on up to 90,000 processor cores in the cloud with Microsoft Azure Virtual Machine Scale Sets, HBv2 virtual machines, and Avere vFXT for Azure. It takes 80 percent off those rendering costs with Azure Spot Virtual Machines and uses an Azure ExpressRoute connection to minimize latency while more securely managing storage in one place, without replication.

Van Gogh Museum

In an average year, tens of thousands of Chinese visitors head to the Netherlands to tour the Van Gogh Museum. In recognition of this interest, the museum has created a mini-program for the social media platform WeChat. Through this app-like program, Chinese users can take selfies and have them digitally reworked in the style of Vincent Van Gogh. To remain compliant with GDPR legislation, and to scale the mini-program to meet demand, the museum has deployed it in Microsoft Azure with Azure Blob Storage, Azure Functions, and Azure Kubernetes Service (AKS).

Wildlife Protection Agency

Wildlife Protection Solutions (WPS) sits in the nexus between the most remote places in the world and burgeoning animal conservation science. It provides the monitoring technology that conservation groups need to keep watch over wild places and protect wildlife. Conservationists use remote cameras to gather image data about the status of the species they protect, but the number of images that must be analyzed before action can be taken is overwhelming. WPS overcomes this barrier in collaboration with Microsoft AI for Earth, supported by Azure technologies for a species-preserving solution.

Learn more

For more details see the FX-series announcement Azure blog, "Optimize extreme computing performance with Azure FX-series Virtual Machines."
Read our recent blog post, "How Azure FX VM Makes Ansys RedHawk-SC™ Run Faster the Less You Spend."
Watch our newest FX-series announcement video.
Read our recent blog post, "Azure HPC After Hours Social Event at ISC 2021."

Deploy apps seamlessly with Red Hat JBoss EAP on Azure App Service—now generally available

Jason Freeberg — Wed, 23 Jun 2021 08:00:28 Z

Red Hat JBoss Enterprise Application Platform (EAP) on Azure App Service is a fully managed offering for Java EE (Jakarta EE) applications. Java developers have been deploying and scaling their Tomcat, Spring Boot, and other Java apps on App Service since 2014—when the platform first supported Java. Since then, our customers have told us they want the same hosting option for their Java EE (Jakarta EE) applications.

We heard your feedback and partnered with Red Hat to offer JBoss EAP on App Service, which has been in preview since September 2020. Our customers are excited to have a managed platform for their Java EE (Jakarta EE) applications and the peace of mind that comes with joint support from Red Hat and Microsoft. Today, the offer is generally available, so you can deploy apps into Red Hat JBoss EAP on App Service with no Red Hat subscription or licenses required.

“Red Hat and Microsoft are both committed to extending choice and flexibility for organizations as they shift traditional Java applications to the cloud. Red Hat JBoss EAP on Azure offers customers a fully configured, jointly supported solution to make it easier for organizations to realize the benefits of cloud-based architecture. We look forward to continuing to build upon our long-standing relationship with Microsoft and expand support for our joint customers."—Rich Sharples, Senior Director of Product Management – Application Services, Red Hat

Figure 1. App service creation App Service interface showing the available options for creating a JBoss EAP application.

This means you can leverage all of App Service’s platform features for your Java EE (Jakarta EE) applications, such as:

Automatic scaling based on metrics such as requests, CPU, memory utilization, or simply the time of day.
Application insights auto-instrumentation for monitoring slow or failing requests, exception logging, and distributed tracing.
Automatic updates for the Operating System, Java virtual machine, and Red Hat JBoss EAP server. You can also choose specific versions of JBoss and update them on your own terms.
Deployment APIs and integrations with Maven, GitHub Actions, and popular IDEs.
Networking integrations to secure and isolate your applications from the internet.

Purpose-built APIs to deploy and run your Java EE applications

Red Hat JBoss EAP on App Service is a jointly supported Pay-as-You-Go (PAYG) offer, which means you do not have to purchase a support subscription from Red Hat ahead of time. Simply create a JBoss EAP web app from the Azure portal or command-line interface (CLI) and start deploying your Java EE apps. If you run into an issue or have a question, create a support case with Microsoft or Red Hat, and support engineers from both companies will collaborate to resolve your case. A support fee for integrated support will automatically be applied to all JBoss EAP sites starting on August 1, 2021*. For more information, check out the App Service pricing page.

Whether your Java applications are packaged in an EAR, WAR, or JAR, App Service has purpose-built APIs to deploy and run your applications. These are integrated into the Azure CLI, Maven plugin, and other tools for a seamless developer experience.

Figure 2. High-level architecture diagram showing Red Hat JBoss EAP on App Service connecting to other Azure services and users.

Red Hat JBoss EAP on App Service is offered on the Premium V3 and Isolated V2 App Service plan types. This means you can deploy Red Hat JBoss EAP onto the latest version of the App Service environment for added security and scale, or onto the multi-tenant variant of App Service. In either case, you can save on your hosting bill with Reserved Instance (RI) pricing, which allows you to make a one- or three-year commitment and save up to 35 percent and 55 percent, respectively.

Ready to get started?

Try the quickstart to learn how to use the Maven plugin to create a Red Hat JBoss EAP site and deploy a sample application. Next, head to Microsoft Learn to connect your JBoss EAP site to a MySQL database running on Azure. Finally, read the migration guide when you are ready to migrate your Java EE (Jakarta EE) applications to Red Hat JBoss EAP on App Service.

Resources

Overview of Red Hat JBoss EAP on Azure.
Migrate a Java EE app to App Service training on GitHub.
Deploy app and dependencies in one step with the Maven plugin for Azure Web Apps.

*Red Hat JBoss EAP Integrated Support is $0.15 per core, per hour and will be eligible for one-year Reserved Instance pricing later this year.

Modernize large-scale NFS workloads and eliminate data silos with Azure Blob Storage

Yuan Zheng — Tue, 22 Jun 2021 11:00:15 Z

Azure Blob Storage, our object storage platform for storing large-scale data helps customers run their applications using a wide range of protocols. With multi-protocol support, customers can run their applications on a single storage platform with no application rewrites necessary, therefore eliminating data silos.

Today, we are announcing that Network File System (NFS) 3.0 protocol support for Azure Blob Storage is now generally available.

Hundreds of enterprise customers from various industries such as manufacturing, media, life science, financial services, and automotive have embraced this feature during our preview and are deploying their workloads in production. These customers have been using NFS 3.0 for a wide array of workloads such as high-performance computing (HPC), analytics, and backup.

Migrate your existing applications using NFS protocol

With this feature, your Azure Blob Storage data can be mounted through NFS 3.0, and accessed through Blob REST API, and analyzed with the Azure Data Lake Service (ADLS) Gen2 API (Spark or Hadoop jobs, for example). Once data is ingested, you can run your existing business-critical NFS workloads such as financial risk simulation, analytics, media processing, and throughput-sensitive HPC jobs.

Samsung Electronics migrated their business innovation center platform which provides a holistic analysis of sales and marketing data across the globe to Azure.

“NFS 3.0 support on Azure Blob Storage linked our existing legacy system to the new modernized data lake so we can provide data insights from various sources. We no longer need to maintain different storage solutions and have eliminated data silos.”—KyungEun Lee, Principal Professional, Samsung Electronics

Run large-scale applications at object storage economics

Azure Blob Storage is the only storage platform that supports NFS 3.0 protocol over object storage natively (no gateway or data copying required), with object storage economics. One of the biggest suppliers in the automotive industry explains:

“To support the development of software in the domain of automated driving and driver assistance, we need to process and store a large amount of data from our development fleet. Azure Blob Storage with NFS 3.0 support helps us to shorten the feedback loop by running large scale simulations faster, more efficient and as a consequence at a lower cost.”

HPCC Systems tested their open-source THOR workload with this feature with hundreds of workers running Azure H series Virtual Machine (standard_h8) and reached 200Gb/s throughput on a single blob storage account.

“It is great to see NFS 3.0 being supercharged with cloud scale. Our THOR workload scales linearly to the object storage limits to generate insights for data from different data sources.”—Flavio Villanustre, VP of Technology and CISO, LexisNexis Risk Solutions Group

The data stored in your storage account with NFS support is billed at the same rate as blob storage capacity charges with no minimal provision capacity required. The flexible pay-as-you-go pricing model gives you an extremely low total cost of ownership (TCO). Learn more about Azure Blob Storage pricing.

Take advantage of flexible data management capabilities

Workloads and applications such as HPC, machine learning, and big data analytics can now take advantage of Azure Blob Storage's large-scale throughput without any additional development required. Customers can choose any tier of Azure Blob Storage and with flexible lifecycle management capabilities, you can move data between tiers based on your needs. TomTom, the location technology specialist, adopted this feature for this reason.

“NFS support has helped us to provide the most accurate traffic insights to our customers seamlessly. NFS 3.0 for Azure Blob Storage provides us both with low latency premium tier and with low-cost cool tiers so that we can lift and shift all our traffic insight workloads to the cloud, and continue to support road authorities, local governments, and drivers with real-time and historical data to help reduce emissions and congestion.”—Eric Bowman, SVP Engineering, TomTom

Get Started

We are excited to help you modernize your NFS workloads with Azure Blob Storage. To get started, check out this video on introducing NFS 3.0 support for Azure Blob Storage and read more about the NFS 3.0 protocol support in Azure Blob Storage.

Optimize extreme computing performance with Azure FX-series Virtual Machines

Andy Jia — Tue, 22 Jun 2021 09:00:14 Z

Today, we are announcing the general availability of the Azure FX-series Virtual Machines available in three regions. Azure FX-series Virtual Machines—based on the 2nd Generation Intel Xeon Scalable Processor—feature a high-performing central processing unit (CPU) clock speed per single core of up to 4 GHz (all core turbo), 21 GB memory per vCPU, and local temporary SSD disks. The high-frequency CPU and memory capacity for FX-series Virtual Machines is designed for workloads demanding very high single-threaded and lightly threaded performance, such as Electronic Design Automation (EDA).

Azure FX-series Virtual Machines are now generally available in the West US 2, West Europe, and Japan East regions, and will come to more Azure regions to meet customer demand.

Fast and cost-efficient FX virtual machine for EDA workloads

EDA is a collection of complex software tools and workflows to design semiconductor integrated circuits. EDA customers have several key challenges:

EDA simulations are compute-intensive jobs with each simulation running from hours to days depending on the complexity of the design. Additionally, some EDA processes are single-threaded. As a result, customers need fast CPU clock speed and a large memory footprint to accelerate their silicon design into fabrication and testing.
EDA software license costs can be several times that of infrastructure costs and often licensed per CPU core. Consequently, EDA users want to lower their licensing costs by maximizing the performance of each CPU core, thereby enabling them to use a smaller number of CPU cores.
EDA designs usually start with different groups working on smaller subsets of a larger design. Small subsets and blocks then need to be merged for additional verification checks at multiple points throughout the process. That verification run, especially in back-end design, demand a high memory footprint.

The Azure FX-series Virtual Machine is designed to solve these EDA workload challenges. The series is powered by Intel Xeon Scalable Processors with a single-core frequency of up to 4.1 GHz, with a base frequency of 3.4 GHz, and an all-core-turbo frequency of 4.0GHz. It delivers up to 19 percent better performance per CPU core over current generation Compute Optimized Virtual Machines. When customers need additional per-CPU core performance, they can turn off hyper-threading by contacting the customer support team. With 1 TB of total memory and 21 GB RAM per virtual CPU core, FX-series Virtual Machines can meet a majority of front-end and back-end design needs. In addition, all the FX-series Virtual Machine sizes come with a local SSD so that memory can read data locally instead of accessing a slower remote Network File System (NFS).

To demonstrate FX-series Virtual Machine performance on the real EDA workloads, we ran an EDA backend physical verification task on E48sv4 and F48mds separately. It took 12.7 hours to complete the workload on E48sv4 versus 11.5 hours on F48mds Virtual Machine. In short, FX-series Virtual Machines shorten the EDA simulation time by 10 percent.

FX for other compute-intensive workloads

Most workloads that require fast CPU frequency and high memory capacity will reap performance benefits from the new FX-series Virtual Machine. The chart below shows the Coremark CPU benchmark results comparing the new FX-series Virtual Machine with Fsv2, both with 48vCPU. FX-series Virtual Machine achieved a 19 percent higher benchmark score.

Empowering customers with significant performance gains

“From the smallest projects to the largest, the performance of Redhawk-SC on Azure’s FX virtual machine family has captivated our interest. Our customers are keen to take advantage of flexible HPC for this task, and Microsoft has been a fantastic partner in delivering it.”—John Lee, General Manager and Vice President, Semiconductor Business Unit, Ansys, Inc

“Intel is proud to partner with Microsoft Azure on their EDA instance. With Intel® Xeon Scalable Processor this new EDA instance that will provide their customers a high-frequency solution to shorten their design cycles."—Trish Damkroger Vice President and General Manager of High Performance Computing, Intel Corporation

Learn more

•    Find out more about high-performance computing in Azure.
•    FX-series Virtual Machine product specification.
•    FX-series Virtual Machine pricing page.
• Read additional details on how FX-series Virtual Machine helps Ansys.
•    Watch the FX-series announcement video.