topic Re: dbutils conflicts with a custom spark extension in Data Engineering

dbutils conflicts with a custom spark extension

maaaxx — Sun, 05 Mar 2023 21:42:22 GMT

Hello dear community,

we have installed a custom spark extension to filter the files allowed to be read into the notebook. It was all good if we use the spark functions.

However, the files are not filtered properly if the user would use e.g., dbutils.fs.cp.

Anyone has an idea why dbutils does not consider the spark extension in this scenario?

Many thanks!

Cheers,

Max

Re: dbutils conflicts with a custom spark extension

Tayyab_Vohra — Mon, 06 Mar 2023 09:28:14 GMT

Hello @Yuan Gao ,

On Databricks, spark and dbutils are automatically injected only into the main entrypoint - your notebook, but they aren't propagated to the Python modules. With spark solution is easy, just use the getActiveSession function of SparkSession class (as SparkSession.getActiveSession()), but you need to continue to pass dbutils explicitly until you don't abstract getting dbutils into some function

The documentation for Databricks Connect shows an example of how it could be achieved. That example has SparkSession as an explicit parameter, but it could be modified to avoid that completely, with something like this:

def get_dbutils():
  from pyspark.sql import SparkSession
  spark = SparkSession.getActiveSession()
  if spark.conf.get("spark.databricks.service.client.enabled") == "true":
    from pyspark.dbutils import DBUtils
    return DBUtils(spark)
  else:
    import IPython
    return IPython.get_ipython().user_ns["dbutils"]

and then in your function, you can use the main function to get the spark dbutils functionality

Re: dbutils conflicts with a custom spark extension

Vartika — Mon, 03 Apr 2023 10:08:45 GMT

Hi @Yuan Gao,

Checking in. If @tayyab vohra's answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?

Thanks!

Re: dbutils conflicts with a custom spark extension

maaaxx — Mon, 10 Apr 2023 13:09:55 GMT

This is the best answer. Thank you. Am Mo., 3. Apr. 2023 um 12:08 Uhr schrieb Databricks Community < community@databricks.com>:

Re: dbutils conflicts with a custom spark extension

Debayan — Mon, 06 Mar 2023 19:31:39 GMT

Hi, could you please explain a little more about the custom spark extension?

Also please tag @Debayan with your next response which will notify me, Thank you!