Hi,

I have not tried it but based on the doc you have to go by this approach. ./environment/bin/pythonmust be replaced with the correct path.

import os
from pyspark.sql import SparkSession

os.environ['PYSPARK_PYTHON'] = "./environment/bin/python"
spark = SparkSession.builder.config(
    "spark.archives",  # 'spark.yarn.dist.archives' in YARN.
    "pyspark_conda_env.tar.gz#environment").getOrCreate() 

In the post, one of the reply does explain The env_dir variable is set to the directory where the conda environment is unpacked by Spark. 

https://community.databricks.com/t5/machine-learning/how-to-use-spark-submit-python-task-with-the-us...