topic Re: How to import a helper module that uses databricks specific modules (dbutils) in Data Engineering

How to import a helper module that uses databricks specific modules (dbutils)

mjbobak — Fri, 09 Sep 2022 01:31:52 GMT

I have a main databricks notebook that runs a handful of functions. In this notebook, I import a helper.py file that is in my same repo and when I execute the import everything looks fine. Inside my helper.py there's a function that leverages built-in dbutils. Now back in my main notebook, when I try to execute the helper function that uses dbutils, I get an error: [NameError: name 'dbutils' is not defined]. How can I create a helper module that imports seamlessly and can leverage dbutils?

Re: How to import a helper module that uses databricks specific modules (dbutils)

mjbobak — Fri, 09 Sep 2022 13:49:15 GMT

Looks like if I add the appropriate imports into the helper.py file then all is corrected.

from pyspark.dbutils import DBUtils
from pyspark.sql import SparkSession
 
spark = SparkSession.builder.getOrCreate()
dbutils = DBUtils(spark)

Re: How to import a helper module that uses databricks specific modules (dbutils)

Atanu — Tue, 13 Sep 2022 18:42:15 GMT

So the above resolved the issue? Please let us know if you still stuck. Thanks

Re: How to import a helper module that uses databricks specific modules (dbutils)

mjbobak — Tue, 13 Sep 2022 18:43:41 GMT

All set. issue resolved

Re: How to import a helper module that uses databricks specific modules (dbutils)

amitca71 — Sun, 11 Dec 2022 15:51:48 GMT

Hi,

i 'm facing similiar issue, when deploying via dbx.

I have an helper notebook, that when executing it via jobs works fine (without any includes)

while i deploy it via dbx (to same cluster), the helper notebook results with

dbutils.fs.ls(path)

NameError: name 'dbutils' is not defined

(for main notebook, that callse the helper function notebook, i have dbutils.widgets, and it doesnt have any issue)

(dbx execute my-task --task=silver --cluster-name="my-multi-cluster": builds a wheel and deploy on the databricks cluster)

adding the includes suggesetd dont resolve the issue.

any advise?

thanks,

Amir

Re: How to import a helper module that uses databricks specific modules (dbutils)

Shawn_Eary — Thu, 31 Aug 2023 20:51:12 GMT

This is a little off topic, but I'm trying to run a PySpark script in VSCode via DataBricks ConnectV2:
https://www.youtube.com/watch?v=AP5dGiCU188
When I do that, I get the error mjbobak describes about dbutils not being defined.

When I use mjbobak's code or the code Elisabetta shares on SO
https://stackoverflow.com/questions/50813493/nameerror-name-dbutils-is-not-defined-in-pyspark
the error goes away, but then I get a runtime error:
"No operations allowed on this path" in response to the following dbutils.fs.ls call:
theFiles = dbutils.fs.ls("/Volumes/myTestData/shawn_test/staging/inbound")

Is there a proper way to define/import dbutils when using Connect V2 to try to debug a PySpark file that is saved locally?