โ08-10-2023 01:06 AM
I have the following command that runs in my databricks notebook.
spark.conf.get("spark.databricks.clusterUsageTags.managedResourceGroup")
I have wrapped this command into a function (simplified).
def get_info():
return spark.conf.get("spark.databricks.clusterUsageTags.managedResourceGroup")
I have then added this function in a .py module, that I install as a private package in the environment of my workspace. I am able to import this function and call it.
However, when I run this function, I receive an error message.
get_info()
>>> NameError: name 'spark' is not defined
If I define the same function in the body of the notebook, I can run it without problems.
- Why bringing this function to a separate module forces me to import spark? What's the proper way of creating a separate module with spark functions? How to import them?
- If possible, what is happening under the hood, that makes it work when I define the function in the notebook, but not work when I import it?
โ08-10-2023 09:27 PM
Hi @daniel23 ,
The behaviour you're experiencing is related to how the spark
object is scoped and available within different contexts in Databricks. When you define and run code directly in a Databricks notebook, the spark
object is automatically available, allowing you to access Spark configuration and features without any additional steps. However, when you define the function in an external module and import it, the scope of the spark
object changes, leading to the "NameError: name 'spark' is not defined" issue.
Here's why this happens and how you can properly create a separate module with Spark functions:
Scope and Context:
spark
object is automatically available in the global scope. When you define a function in the notebook itself, it can directly access the spark
object because it's defined in the same notebook context.spark
object. This is why you encounter the "NameError" when the function is imported and executed.Proper Approach:
spark
object as an argument to the functions in the module. This way, the function in the module knows where to find the spark
object.โ08-10-2023 09:27 PM
Hi @daniel23 ,
The behaviour you're experiencing is related to how the spark
object is scoped and available within different contexts in Databricks. When you define and run code directly in a Databricks notebook, the spark
object is automatically available, allowing you to access Spark configuration and features without any additional steps. However, when you define the function in an external module and import it, the scope of the spark
object changes, leading to the "NameError: name 'spark' is not defined" issue.
Here's why this happens and how you can properly create a separate module with Spark functions:
Scope and Context:
spark
object is automatically available in the global scope. When you define a function in the notebook itself, it can directly access the spark
object because it's defined in the same notebook context.spark
object. This is why you encounter the "NameError" when the function is imported and executed.Proper Approach:
spark
object as an argument to the functions in the module. This way, the function in the module knows where to find the spark
object.โ08-14-2023 10:46 AM
Thanks for your reply.
I have redefined the function, including spark in the scope:
def get_info(spark: SparkSession):
return spark.conf.get("spark.databricks.clusterUsageTags.managedResourceGroup")
After implementing the change, it works.
Hence, thank you for the explanation and the suggested approach.
Best,
โ08-15-2023 08:59 PM
Awesome @daniel23 !
I'm glad it helped.
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!