Easy Install Pyspark in Anaconda

Install on Mac

Pre Installations:

Anaconda : https://docs.anaconda.com/anaconda/install/mac-os/

Step by Step to Install PySpark

1. Install Java

2. Install pyspark

3. Install findspark

4. Test Spark Installation

5. Launch Anaconda

6. Launch Jupyter Lab

7. Spark Commands

Step 1: Install Java

conda install openjdk

Step 2: Install pyspark

conda install pyspark

Step 3: Install findspark

conda install -c conda-forge findspark

Step 4: Test Spark Installation

In command prompt

pyspark

data_values = [('Apple',3),('Banana',6),('Orange', 9)]
column_name = ['Name', 'Count']
df = spark.createDataFrame(data_values).toDF(*column_name)
df.show()

Step 5: Launch Anaconda

Step 6: Launch Jupyter Lab

http://localhost:8888/lab

Step 7: Spark Commands

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('pySparkSetup').getOrCreate()

data_values = [('Apple',3),('Banana',6),('Orange', 9)]
column_name = ['Name', 'Count']
df = spark.createDataFrame(data_values).toDF(*column_name)
df.show()

Stay tuned for next episodes for Pyspark

References

https://sparkbyexamples.com/pyspark/install-pyspark-in-anaconda-jupyter-notebook/

Introduction and Installation of PySpark for Mac Users

How to Install PySpark and Apache Spark on MacOS - Luminis

Complete Guide to Installing PySpark on MacOS