Multiple Workers in a Single Node Configuration for Spark Standalone Cluster

A few days back I tried to set up a spark standalone cluster in my own machine with the following specification: two workers (balanced cores) within a single node.

Here’s how you’d do it.

Step 1. Start the master instance via <path_to_sbin>/start-master.sh

Step 2. Configure the spark environment properties.

Go to the conf directory and add the following lines to spark-env.sh.

export SPARK_WORKER_INSTANCES=<number_of_workers>
export SPARK_WORKER_CORES=<total_cores_for_the_workers>

The property names should be intuitive. We specified the number of worker instances and the total number of cores for the workers.

Step 3. Start the slave node via <path_to_sbin>/start-slave.sh <path_to_master>

Step 4. To check the configuration results, just go to <your_host_address>:<spark_master_webui_port>.