If you hit an issue related to an already existing Spark context, you can only have one SparkContext instance in a single JVM. In such cases, you can try the following approach

from pyspark import SparkContext
from pyspark.sql import SparkSession

# Check if a Spark context already exists
try:
    sc = SparkContext.getOrCreate()
    spark = SparkSession(sc)
    print("Using existing Spark context.")
except Exception as e:
    print("No existing Spark context found. Creating a new one.")
    sc = SparkContext()
    spark = SparkSession(sc)

Thanks for your responses.
I did try creating a spark context and feed it to create a spark session constructor, but I get an error from databricks which states I should not initialize a new context despite using SparkContext.getOrCreate() Method.
Tried the following as well:

1. SparkSession.getActiveSession() (returns null)
   
   
2. getOrCreate method on SparkContext and SparkSession ( Asks for Master URL and app name, after which it states the databricks error: Should not intialize session or context in databricks if one already exists.)

Is there some solution for this.We got struck where a cluster having unity catalog is not able to get spark context.This is not allowing to use distributed nature of spark in databricks.