Understanding HDInsight Spark jobs and data through visualizations in the Jupyter Notebook

The Jupyter Notebook on HDInsight Spark clusters is useful when you need to quickly explore data sets, perform trend analysis, or try different machine learning models. Not being able to track the status of Spark jobs and intermediate data can make it difficult for data scientists to monitor and optimize what they are doing inside the Jupyter Notebook.

To address these challenges, we are adding cutting edge job execution and visualization experiences into the HDInsight Spark in-cluster Jupyter Notebook. Today, we are delighted to share the release of the real time Spark job progress indicator, native matplotlib support for PySpark DataFrame, and the cell execution status indicator.

Spark job progress indicator

When you run an interactive Spark job inside the notebook, a Spark job progress indicator with a real time progress bar appears to help you understand the job execution status. You can also switch tabs to see a resource utilization view for active tasks and allocated cores, or a Gantt chart of jobs, stages, and tasks for the overall workload.

Native matplotlib support for PySpark DataFrame

Previously, PySpark did not support matplotlib. If you wanted to plot something, you would first need to export the PySpark DataFrame out of the Spark context, convert it into a local python session, and plot from there. In this release, we provide native matplotlib support for PySpark DataFrame. You can use matplotlib directly on the PySpark DataFrame just as it is in local. No need to transfer data back and forth between the cluster spark context and the local python session.

Cell execution status indicator

Step-by-step cell execution status is displayed beneath the cell to help you see its current progress. Once the cell run is complete, an execution summary with the total duration and end time will be shown and kept there for future reference.

Getting started

These features have been built into the HDInsight Spark Jupyter Notebook. To get started, access HDInsight from the Azure portal. Open the Spark cluster and select Jupyter Notebook from the quick links.

Feedback

We look forward to your comments and feedback. If you have any feature requests, asks, or suggestions, please send us a note to cosctcs@microsoft.com. For bug submissions, please open a new ticket.

For more information, check out the following:

• Kernels for Jupyter notebook on Spark clusters in Azure HDInsight