topic Re: how to get dbutils in Runtime 13 in Data Engineering

how to get dbutils in Runtime 13

grazie — Fri, 19 May 2023 12:54:57 GMT

We're using the following method (generated by using dbx) to access dbutils, e.g. to retrieve parameters from secret scopes:

    @staticmethod
    def _get_dbutils(spark: SparkSession) -> "dbutils":
        try:
            from pyspark.dbutils import DBUtils  # type: ignore
 
            if "dbutils" not in locals():
                utils = DBUtils(spark)
                return utils
            else:
                return locals().get("dbutils")  # type: ignore
        except ImportError:
            return None  # type: ignore

However, I discovered when running on Databricks Runtime 13 that this no longer works.

I checked release notes for 13, but saw nothing relevant.

How can we access dbutils in library code in runtime 13?

Re: how to get dbutils in Runtime 13

User16463165548 — Mon, 22 May 2023 23:45:22 GMT

Hi, please use a Machine Learning DBR (Databricks Runtime X.Y for Machine Learning) that comes with DBUtils preinstalled:

Databricks Runtime 13.0 for Machine Learning

Re: how to get dbutils in Runtime 13

colt — Tue, 23 May 2023 14:50:56 GMT

We have something similar in our code. This worked using runtime 13 until last week. Also the Machine Learning DBR doesn't work either.