Streamlining Spark 3.5.1 Installation: A Step-by-Step Guide

Spark in the realm of computing, symbolizes innovation and efficiency. It’s not merely a word; it’s a beacon of transformative technology. Spark encapsulates the essence of rapid data processing, real-time analytics, and scalable solutions. Like its namesake, Spark ignites the imagination of developers and data scientists, empowering them to harness the power of big data with unprecedented speed and agility. It represents a spark of ingenuity that fuels the fire of progress in the ever-evolving landscape of data-driven decision-making.

Spark-3.5.1 Installation
————————————————————————————————-
: INSTRUCTIONS :

[ You can change /home/divithraju/Downloads/path/ use your path location on your machine ]
1. Download Hadoop and Java
tar -zxvf spark-3.5.1.tar.gz (Extract the tar file)
tar -zxvf jdk-8u391-linux-x64.tar (Extract the tar file)

sudo apt-get install vim (Install USER Friendly Editer)

vi ~/.bashrc (Set the java Path in your Home Path)
export JAVA_HOME=/home/divithraju/Downloads/jdk-8u391-linux-x64/jdk1.8.0_391
export SPARK_HOME=/home/divithraju/Downloads/spark-3.5.1-bin-hadoop3
export PATH=$HOME/bin:$JAVA_HOME/bin:$SPARK_HOME/bin:$PATH

source ~/.bashrc
echo $JAVA_HOME   # Check if JAVA_HOME is set correctly

[ You can change /home/divithraju/Downloads/path/ use your path location on your machine ]
[ And don’t change any orther shell ]
=============================

2.Start Spark
cd ~/Downloads/spark-3.5.1-bin-hadoop3
ls -a
cd conf  [ open file ]
ls
cp spark-env.sh.template spark-env.sh  [ Copy the file  ]
vi spark-env.sh
export JAVA_HOME=/home/divithraju/Downloads/jdk-8u391-linux-x64/jdk1.8.0_391  [ Add this comment in file ]
cd ..
bin/spark-shell [Scala]
bin/pyspark [ Python ]
sbin/start-all.sh [ start All spark Related Services ]
http://<spark-master-host>:8080 [UI]

Stop All Spark Related Services
sbin/stop-all.sh

“Thank you for reading! If you enjoyed this article and want to stay updated on my latest insights and projects, feel free to connect with me on LinkedIn.”