How to properly import spark functions?

[daniel23]

I have the following command that runs in my databricks notebook.

spark.conf.get("spark.databricks.clusterUsageTags.managedResourceGroup")

I have wrapped this command into a function (simplified).

def get_info():
    return spark.conf.get("spark.databricks.clusterUsageTags.managedResourceGroup")

I have then added this function in a .py module, that I install as a private package in the environment of my workspace. I am able to import this function and call it.

However, when I run this function, I receive an error message.

get_info()
>>> NameError: name 'spark' is not defined

If I define the same function in the body of the notebook, I can run it without problems.

- Why bringing this function to a separate module forces me to import spark? What's the proper way of creating a separate module with spark functions? How to import them?

- If possible, what is happening under the hood, that makes it work when I define the function in the notebook, but not work when I import it?

[Kaniz_Fatma · Accepted solution]

Hi @daniel23 , 

The behaviour you're experiencing is related to how the spark object is scoped and available within different contexts in Databricks. When you define and run code directly in a Databricks notebook, the spark object is automatically available, allowing you to access Spark configuration and features without any additional steps. However, when you define the function in an external module and import it, the scope of the spark object changes, leading to the "NameError: name 'spark' is not defined" issue.

Here's why this happens and how you can properly create a separate module with Spark functions:

1. Scope and Context:

   • In a Databricks notebook, the spark object is automatically available in the global scope. When you define a function in the notebook itself, it can directly access the spark object because it's defined in the same notebook context.
   • However, when you define the function in an external module, it loses access to the notebook's global scope, including the spark object. This is why you encounter the "NameError" when the function is imported and executed.

2. Proper Approach:

   • To create a separate module with Spark functions that can be imported and used, you need to explicitly pass the spark object as an argument to the functions in the module. This way, the function in the module knows where to find the spark object.

[daniel23]

Thanks for your reply.

I have redefined the function, including spark in the scope:

def get_info(spark: SparkSession):
    return spark.conf.get("spark.databricks.clusterUsageTags.managedResourceGroup")

After implementing the change, it works.

Hence, thank you for the explanation and the suggested approach.

Best,

[Kaniz_Fatma]

Awesome @daniel23 !

I'm glad it helped.