Apache Spark On Apple Silicon

Install Spark(Scala, Python) on Macs having M1, M1 Pro, M1 Max, and M2 with this guide

Photo by Sumudu Mohottige on Unsplash

Apple Silicon is the processor architecture used inside Apple’s computer chips. The likes of the M1 and M2 processors both use Apple Silicon, marking a departure from Apple’s use of Intel CPUs for a decade.

The easiest way to install apple silicon is as follows:

1. Install Homebrew:

Homebrew is a free and open-source software package management system that simplifies the installation of software on Apple’s operating system, macOS, as well as Linux. The name is intended to suggest building software on the Mac depending on the user’s taste. It can be installed by running the following command in the terminal

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

The installation prompts you to give your mac password as it requires admin privileges to install stuff on your mac. After the installation is done the setup starts you to run a few commands which are shown as follows

echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> /Users/admin/.zprofile
eval "$(/opt/homebrew/bin/brew shellenv)"

2. Install Java:

Spark requires Java and is required to be installed and it can be done using homebrew via the command in the terminal as shown below. This step can be ignored if Java is already present on your mac.

brew install openjdk@11

3. Install Scala:

Scala is a strong statically typed general-purpose programming language that supports both object-oriented programming and functional programming. Apache Spark as you all might be aware is written in Scala and it can be installed via the command in the terminal as shown below

Note: This step can be ignored if you want to install PySpark rather than scala.

brew install scala

4. Install Python:

Folks who want to install PySpark as well need to install python as well and it can be installed via homebrew using the command. This step can be ignored if you already have python installed on your mac

brew install python

5. Install Spark:

To install Apache Spark through the command line, enter and run the following command in the Terminal :

brew install apache-spark

6. Verify Spark — Scala:

The installation takes a few minutes and once it's done verify the installation of Spark via the following command in the terminal.

spark-shell

The command would start up spark and the console would look something like the following

7. Verify Spark — Python:

To Verify if PySpark has been installed, run the following command in the terminal.

pyspark

The command would start up PySpark and the console would look something like the following

And thus you have installed Spark(Python & Scala) Successfully on Apple Silicon Mac.

Clap if you like this story, and follow me on Linkedin and Medium for more Interesting stuff. Cheers !!!

Check out my other stories as well. 😊