topic Re: Databricks Python Import Bug in Data Engineering

Databricks Python Import Bug

alvaro_databric — Mon, 08 May 2023 12:21:34 GMT

Hello,

I am reaching to the community in order to shed some light into a Bug I have been encountering recently. The two setups are as follows:

SETUP-1 (WORKS):

Python Notebook in Workspace FileSystem (this is Shared/folder/...)
Python custom wheel library in .whl installed into the cluster

SETUP-2 (DOES NOT WORK)

Python Notebook in Repos
Same wheel and cluster as SETUP-1

Moreover SETUP-2 is able to improt some of the functions but not all of them. At first I thought it was an issue with the Wheel generation, but SETUP-1 works just fine and is able to improt everything.

This issue makes me think that there might be a difference into how Databricks manages filesystem or any other variables in Repos that I am not grasping.

Can someone indicate me what could be the issue here or any important difference when working with databricks Repos?

Re: Databricks Python Import Bug

Anonymous — Fri, 19 May 2023 08:31:47 GMT

Hi @Alvaro Moure

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Databricks Python Import Bug

Debayan — Wed, 10 May 2023 08:30:11 GMT

Hi,

How can I run non-Databricks notebook files in a repo?

For example, a .py file?

You can use any of the following:

Bundle and deploy as a library on the cluster.
Pip install the Git repository directly. This requires a credential in secrets manager.
Use
%run
with inline code in a notebook.
Use a custom container image. See Customize containers with Databricks Container Services.

You can refer to: https://docs.databricks.com/repos/limits.html and https://docs.databricks.com/repos/limits.html#non-notebook-files-files-in-repos

Also, refer to the errors: https://docs.databricks.com/repos/errors-troubleshooting.html

Please let us know if this helps and let us know the errors. Also, please tag @Debayan with your next comment so that I will get notified. Thank you!

Re: Databricks Python Import Bug

alvaro_databric — Fri, 15 Sep 2023 10:37:36 GMT

Solution has comed with an update, as stated in Work with Python and R modules | Databricks on AWS, In Databricks Runtime 13.0 and above, directories added to the Python sys.path are automatically distributed to all executors in the cluster. In Databricks Runtime 12.2 LTS and below, libraries added to the sys.path must be explicitly installed on executors.

This seems to have solved our strange import problem from Databricks Repos