Creates an HDInsight cluster running Apache Spark 1.4.1.

Azure Public Test Date Azure Public Test Result

Azure US Gov Last Test Date Azure US Gov Last Test Result

Best Practice Check Cred Scan Check Creates HDInsight Linux cluster and run custom script action to install Apache Spark 1.4.1

Deploy To Azure Deploy To Azure US Gov Visualize

This template creates an HDInsight Linux based cluster and then updates the cluster headnodes with the Apache Spark 1.4.1 binaries(including YARN support).
Additionally, it sets specific environment variables ($SPARK_HOME, updates $PATH) to allow for easy access to the Spark client binaries.

Please be sure to utlize appropriate Spark core, memory, and executor settings based on your chosen deployment size.

To launch Spark interactivly, please SSH into the cluster (clustername-ssh.azurehdinsight.net) and execute the following commands:

Sudo -i
$SPARK_HOME/bin/spark-shell

You should see output similar to the following:

15/10/01 15:21:34 INFO util.Utils: Successfully started service 'HTTP class server' on port 47985.
Welcome to version 1.4.1

Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
15/10/01 15:21:40 INFO spark.SparkContext: Running Spark version 1.4.1

...Output Snipped...

scala>

To deploy alternative versions of Apache Spark as published by the HDInsight team, please review follow this link: Click Me

Tags: Microsoft.Network/virtualNetworks, Microsoft.Storage/storageAccounts, Microsoft.HDInsight/clusters