Bundled Wheel Task with Serverless Compute

mlrichmond-mill
New Contributor III

I am trying to run a wheel task as part of a bundle on serverless compute. My databricks.yml includes an artifact being constructed:

artifacts:
  nexusbricks:
    type: whl
    build: python -m build
    path: .

I then am trying to set up a job to consume it:

resources:
  jobs:         
    ingest_file:
      name: "Ingest File Data"

      parameters:
        - name: scriptName        
          default: ""

      tasks:
        - task_key: ingest_file
          environment_key: serverless_env
          python_wheel_task:
            package_name: "nexusbricks"
            entry_point: "ingestRouter"            
            parameters:
              - "--scriptName"
              - "{{job.parameters.scriptName}}"   # sys.argv[1]

      environments:
        - environment_key: serverless_env
          spec:
            environment_version: "2" 
            dependencies:
            - "${workspace.root_path}/artifacts/.internal/nexusbricks-0.1.0-py3-none-any.whl"

 

Eventually, I would like to be able to upload the wheel to a dynamic location based on its git_commit and then use any past version of the wheel in my job - but for right now I am just trying to get this simple example to work.

I know I can make this work using the libraries tag when using a traditional job cluster - but how do I get this to work for serverless?

I haven't found an example of this workflow on the forums so far - the closest I saw was someone using notebooks but notebook tasks that consume a wheel aren't the same thing as a wheel task.