We are excited to announce the general availability of the HDInsight Tool for IntelliJ. The HDInsight Tool for IntelliJ serves the open source community and will be of interest to HDInsight Spark developers. It has seamless integration with IntelliJ and enables users to enjoy the native support for the Scala and Java Spark application development experience.

For users who develop Spark applications on HDInsight, it can sometimes be hard to get started with building their first application, and the development cycle of Spark application from IDE could be long and tedious. With this tool, users can get started with HDInsight Spark in just a few minutes. Experienced Spark developers can iterate their development cycle faster and easier.

Tool overview

The IntelliJ HDInsight tool extends IntelliJ to support Spark job life cycle from create, author, debug, to submit job to an Azure HDInsight Spark cluster. It integrates well with Azure to allow users to navigate HDInsight Spark clusters and to view associated Azure storage account. To further boost productivity, the IntelliJ HDInsight tool also offers the capability to view Spark job history, display detailed job logs, and show the job output. A few usability improvements have been implemented upon user preview feedback, including auto create artifact, locate SDK, cache Spark logs, etc..

The HDInsight Tool for IntelliJ offers the following capabilities:

• Create Spark project with built-in templates, sample code and intelligence to auto create artifact and locate assembly
• Develop Spark applications with native authoring support (e.g. IntelliSense, auto format, error checking, etc.) from IntelliJ (mainly Scala and Java)
• Submit Spark applications to HDInsight clusters
• Integrate with YARN and Spark to view job history, job logs and job results
• Allow user to stop a running Spark application before its completion
• Test and validate the Spark application locally
• View all the Spark clusters associated with user’s azure subscriptions
• Navigate the associated storage resources of user’s HDInsight Spark cluster
• Cache the Spark logs into disk in case the Spark logs are too big
• Remote debug

Create, author, submit and stop a Spark application

The HDInsight Tool for IntelliJ supports Spark application life cycle management from creation to submission. It allows users to view job history, information and statistics for Spark applications running in the cluster with tight integration with IntelliJ and seamless integration with Azure.

Create a new Spark application

View Spark cluster and navigate the associated storage accounts

Submit Spark application

Stop Spark applications from IntelliJ UI

After submitting Spark jobs to HDInsight, the tool allows users to stop a running Spark application in IntelliJ before its completion.

View job history, job output and job log

Installation

User can get the latest bits by going to IntelliJ repository, and searching “HDInsight.” IntelliJ will also prompt users for latest update if user has already installed the plugin.

For more information, check out the following links:

• Video: Introducing HDInsight Tools for IntelliJ for HDInsight Spark Development
• Documentation: Use HDInsight Tools Plugin for IntelliJ IDEA to create and submit Spark Scala applications (Linux)
• Learn more about today’s announcements on the Azure Data Lake Blog
• Discover more Azure service updates

If you have questions, feedback, comments, or bug reports, please use the comments below or send a note to hdivstool@microsoft.com.