topic Re: Databricks Asset Bundles + Artifacts + Poetry in Administration & Architecture

Databricks Asset Bundles + Artifacts + Poetry

Fiabane — Wed, 16 Oct 2024 20:56:43 GMT

Hello,

I've configured the DABs on our project successfully. Moreover, I could switch from setuptools to poetry almost successfully. In the project's databricks.yml I configured it as the documentation suggested, I've just changed the name of the artifact:

artifacts:

config:

type: whl

build: poetry build

path: .

The class that I want to 'compile' as a package/module is in the project's root/config folder:

project/
├── config/
│ ├── __init__.py
│ └── env_manager.py
├── pyproject.toml
├── poetry.lock

Also, in the codepipeline template (AWS), I needed to add the following commands (which were not mentioned in the documentation):

- sudo apt update
- sudo apt install -y pipx
- pipx install poetry
- pipx ensurepath
- export PATH="$PATH:/root/.local/bin"
- poetry --version

The deployment was done (shown in the Figure below).

However, when I tried to run the job I got ModuleNotFoundError: No module named 'config' error message.

The way I am calling it on the files is: import config.env_manager as workspace

Any help would be greatly appreciated.

Re: Databricks Asset Bundles + Artifacts + Poetry

filipniziol — Thu, 17 Oct 2024 07:11:11 GMT

Hi @Fiabane ,

Could you first check:

Do you see your .whl file in your artifacts folder?
Could you try to install the package by running the code in your notebook :

%pip install <path to your wheel>

As far as I understand you want to have a job with a notebook that is using this package.
There is a whole video that covers this scenario:
https://www.youtube.com/watch?v=dbuSAdcagEk

Could you check if your job task has dependent libraries set with the path tyou your .whl file?

Re: Databricks Asset Bundles + Artifacts + Poetry

Fiabane — Thu, 17 Oct 2024 23:32:19 GMT

Hi @filipniziol ,

Thank you very much for answering me.

Yes, I do (wheel_prod.png).

I could also install it locally too

The only part that was missing was to call it on the job.

The sad news is although the package was successfully deployed and called the command dbutils.widget.get('catalog') did not work

def load_env(): """ Load configuration based on the environment. """ try: catalog = dbutils.widgets.get("catalog") except: catalog = 'development'

And the catalog variable was ingested successfully:

Thank you once again for your attention and kindness.