Databricks Community

pablogarcia · a week ago

Hi everyone! I’m working with a job running on Databricks serverless, and I’d like to know how we can load a wheel file that we have stored in a volume, and then use that wheel as a package within the job itself. Any guidance or examples would be appreciated!

szymon_dybczak · Friday

Hi @pablogarcia ,

You need configure serverless environement to achive that. Refere to below documentation:

Configure the serverless environment | Databricks on AWS

Specifically to those sections:

- Configure the serverless environment | Databricks on AWS

pablogarcia · Friday

Hi,

Thanks a lot for the documentation, but we dont configure any job manually in DBX UI everything happens programmatically using yaml and deploy with DBX Bundle. For serverless I'm using

      environments:
        - environment_key: serverless_default
          spec:
            environment_version: "4"
            dependencies:
              - pip:
#                  - "extra-index-url=https://almengineering:${var.azure_pat}@pkgs.dev.azure.com/almengineering/_packaging/MLEngineering/p..."
# - "databricks-mlops-stack==0.2.64"
                  - databricks-feature-engineering==0.12.1
                  - "locust>=2,<3"
                  - "geventhttpclient>=2,<3"
              - whl: "/Volumes/catalog/sanity_build/wheels_volume/databricks_mlops_stack-0.2.63-py3-none-any.whl"

I've tried multiple possible conviction, and with non of them I manage to get the dependency in the serverless. I even create a script to check the dependencies in the serverless when it runs and the dependency is not there.