HDInsight tools June release has been released! In this release, you can choose SBT as a build tool in addition to Maven when creating Spark projects in IntelliJ. With improvements to Spark job view and job graph in IntelliJ and Eclipse, more job info and statistics are now provided. You can also view job logs including driver stderr, stdout, and directory info easily in Spark job view.

Summary of key updates

Improved Spark job view and job graph in IntelliJ and Eclipse

• In Azure Explorer (View > Tools window > Azure Explorer in IntelliJ), go to HDInsight node, select the Spark cluster, and then click Jobs as shown below.

• The left pane of Spark job view shows all the Spark applications that ran in the cluster. Select one Spark job to view more details.

• By hovering over the job graph, it will show the job run information. When clicking on job graph, it dives into the stage graph and shows the statistics of the job. You can also open Spark history UI or Yarn UI by clicking the respective link at the top of the Spark job view.

Improved Spark log view in IntelliJ and Eclipse 

• In the same Spark job view pane, click the Log tab to view the frequently used logs including driver stderr, driver stdout, and directory info, as shown below.

SBT build tool support when creating Spark project in IntelliJ

• As shown below, you can now choose SBT as a build tool in addition to Maven when creating a new Spark project in IntelliJ.

• Following the wizard to create a Spark project, after it is done, a new build.sbt file is generated which contains the build description for the project. Then you can author, submit, or debug the Spark job following the Spark job submission/debugging instructions.

How to install/update

• HDInsight Eclipse plugin: Eclipse will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the Eclipse repository and searching “Azure Toolkit for Java”.

• HDInsight IntelliJ plugin: IntelliJ will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the IntelliJ repository and searching “Azure Toolkit for IntelliJ”.

For more information, check out the following:

• HDInsight Visual Studio plugin (Demo video)
• HDInsight Eclipse plugin (Demo video)
• HDInsight IntelliJ plugin (Demo video)

Learn more about today’s announcements on the Azure blog and Big Data blog, and discover more Azure service updates.

Feedback

We look forward to your comments and feedback. If there is any feature request, customer ask, or suggestion, please send us a note to hdivstool@microsoft.com. For bug submission, please open a new ticket using the template.