topic Re: ModuleNotFoundError: No module named 'pyspark.dbutils' in Data Engineering

ModuleNotFoundError: No module named 'pyspark.dbutils'

vk217 — Mon, 25 Sep 2023 22:23:22 GMT

I have a class in a python file like this

from pyspark.sql import SparkSession from pyspark.dbutils import DBUtils class DatabricksUtils: def __init__(self‌‌): self.spark = SparkSession.getActiveSession() self.dbutils = DBUtils(self.spark) def get_dbutils(self) -> DBUtils: return self.dbutils

In another python file, I am importing this module and calling the db utils like

from .myProject.functions.utils import *

db = DatabricksUtils()

dbutils = db.get_dbutils()

This works when I test it locally in vscode but in Azure Pipeline when I try to build my unit tests fail and I get a message

ModuleNotFoundError: No module named 'pyspark.dbutils'

Re: ModuleNotFoundError: No module named 'pyspark.dbutils'

Miguel_Suarez — Thu, 28 Sep 2023 18:58:14 GMT

Hey @vk217 ,

What Databricks runtime version was the cluster you ran the code on? I was able to successfully run your code in a 12.2 and a 13.3 cluster.

Can you try running it on a cluster with one of those DBR versions, if you haven't already? And please let us know if you're still running into issues.

Best,

Miguel

Re: ModuleNotFoundError: No module named 'pyspark.dbutils'

Poonam_Joshi — Fri, 17 Nov 2023 00:43:23 GMT

We are trying to do similar thing . we are using dbutils to get secret scope and we doing unittesting in azure pipeline where it gives error pyspark.dbutils not found

We have tried using databricks.sdk.dbutils for which I got authentication error li kr value not found I used databricks -connect library as well but still the same issue..

Can you please help with it?

Re: ModuleNotFoundError: No module named 'pyspark.dbutils'

Jarkrung — Thu, 30 May 2024 03:35:33 GMT

Hi, we are also in the same exact situation. Were you able to solve the problem? Or a workaround maybe.

Re: ModuleNotFoundError: No module named 'pyspark.dbutils'

T0M — Wed, 26 Feb 2025 12:27:08 GMT

Had the same Problem in my GitLab CI/CD Pipeline while trying to deploy:

$ databricks bundle deploy -t dev Building package... Error: build failed package, error: exit status 1, output: Traceback (most recent call last): [...] File "/builds/user/package/./src/package/main.py", line 2, in <module> from pyspark.dbutils import DBUtils ModuleNotFoundError: No module named 'pyspark.dbutils'

Solved it by completing the requirements.txt with

ipykernel>=6.29.4
nbformat>=5.10.4
databricks-connect>=13.1.0

as seen here: https://github.com/databricks/dais-cow-bff/blob/dais24-main/requirements.txt