Hi @badari_narayan ,

In general you may run pyspark project locally, but with limitations.

1. Create virtual environment
2. Install pyspark in your virtual environment (the same version you have on your cluster)
3. Since spark version 2.x you even do not need to create a local spark cluster.
   
   

spark = SparkSession.builder \
    .appName("LocalSparkApp") \
    .master("local[*]") \
    .getOrCreate()​

In order to work it locally you will have limitations and also you will need to set it up properly.
For example you will need to create first your spark instance like above (in databricks workspace it is already available). Also, you will be able to run Spark SQL available in pyspark, but not Databricks prioprietary SQL.
Locally, you also do not have a unity catalog configured in your workspace.

Recently Databricks team made unity catalog open source, so you may check the pages below:
https://www.unitycatalog.io/
https://github.com/unitycatalog/unitycatalog

Still, you will need to setup your local unity catalog, if you do not want to connect databricks workspace and then to run the code to create the function in your local unity catalog.