@Aravind A​ , What are the dependencies and versions you installed?

Install pyspark + openjdk

%pip install pyspark==2.4.8
!apt-get install openjdk-8-jdk-headless -qq > /dev/null

Build spark session

from pyspark.sql import SparkSession
spark = SparkSession.builder\
    .appName("Practice”)\
    .getOrCreate()

Note that:

Spark runs on Java 8/11, Scala 2.12/2.13, Python 3.6+ and R 3.5+. 

Python 3.6 support is deprecated as of Spark 3.2.0. 

Java 8 prior to version 8u201 support is deprecated as of Spark 3.2.0. 

For the Scala API, Spark 3.2.1 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x). 

https://spark.apache.org/docs/latest/