dbutils conflicts with a custom spark extension
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-05-2023 01:42 PM
Hello dear community,
we have installed a custom spark extension to filter the files allowed to be read into the notebook. It was all good if we use the spark functions.
However, the files are not filtered properly if the user would use e.g., dbutils.fs.cp.
Anyone has an idea why dbutils does not consider the spark extension in this scenario?
Many thanks!
Cheers,
Max
- Labels:
-
Dear Community
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-06-2023 01:28 AM
Hello @Yuan Gao ,
On Databricks, spark and dbutils are automatically injected only into the main entrypoint - your notebook, but they aren't propagated to the Python modules. With spark solution is easy, just use the getActiveSession function of SparkSession class (as SparkSession.getActiveSession()), but you need to continue to pass dbutils explicitly until you don't abstract getting dbutils into some function
The documentation for Databricks Connect shows an example of how it could be achieved. That example has SparkSession as an explicit parameter, but it could be modified to avoid that completely, with something like this:
def get_dbutils():
from pyspark.sql import SparkSession
spark = SparkSession.getActiveSession()
if spark.conf.get("spark.databricks.service.client.enabled") == "true":
from pyspark.dbutils import DBUtils
return DBUtils(spark)
else:
import IPython
return IPython.get_ipython().user_ns["dbutils"]
and then in your function, you can use the main function to get the spark dbutils functionality
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-06-2023 11:31 AM
Hi, could you please explain a little more about the custom spark extension?
Also please tag @Debayan with your next response which will notify me, Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-03-2023 03:08 AM
Hi @Yuan Gao,
Checking in. If @tayyab vohra's answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?
Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2023 06:09 AM

