Databricks Asset Bundles + Artifacts + Poetry

[Fiabane]

Hello,

I've configured the DABs on our project successfully. Moreover, I could switch from setuptools to poetry almost successfully. In the project's databricks.yml I configured it as the documentation suggested, I've just changed the name of the artifact:

Spoiler

artifacts:

    config:

       type: whl

       build: poetry build

       path: .

The class that I want to 'compile' as a package/module is in the project's root/config folder:

Spoiler

project/
├── config/
│     ├── __init__.py
│     └── env_manager.py
├── pyproject.toml
├── poetry.lock

Also, in the codepipeline template (AWS), I needed to add the following commands (which were not mentioned in the documentation):

Spoiler

- sudo apt update
- sudo apt install -y pipx
- pipx install poetry
- pipx ensurepath
- export PATH="$PATH:/root/.local/bin"
- poetry --version

The deployment was done (shown in the Figure below).

However, when I tried to run the job I got ModuleNotFoundError: No module named 'config' error message.

The way I am calling it on the files is: import config.env_manager as workspace

Any help would be greatly appreciated.

[filipniziol]

Hi @Fiabane ,

Could you first check:

• Do you see your .whl file in your artifacts folder?
• Could you try to install the package by running the code in your notebook :

         %pip install  <path to your wheel>

As far as I understand you want to have a job with a notebook that is using this package.
There is a whole video that covers this scenario:
https://www.youtube.com/watch?v=dbuSAdcagEk

Could you check if your job task has dependent libraries set with the path tyou your .whl file?

[Fiabane · Accepted solution]

Hi @filipniziol ,

Thank you very much for answering me.

Yes, I do (wheel_prod.png).

I could also install it locally too

The only part that was missing was to call it on the job.

The sad news is although the package was successfully deployed and called the command dbutils.widget.get('catalog') did not work

def load_env():
    """
    Load configuration based on the environment.
    """
    try: 
        catalog = dbutils.widgets.get("catalog")
    except:
        catalog = 'development'

And the catalog variable was ingested successfully:

Thank you once again for your attention and kindness.