denis-dbx
Databricks Employee
Databricks Employee

We've added a solution for this problem in v0.245.0. There is opt-in "dynamic_version: true" flag on artifact to enable automated wheel patching that break the cache (Example). Once set, "bundle deploy" will transparently patch version suffix in the built wheel so that the new code is always deployed.

Note, that this only works with local wheels that are uploaded as part of "bundle deploy". So you need to download this wheel into your project so that it can be patched by "bundle deploy" before upload to break the cache.

artifacts:
  mywheel:
    type: whl
    dynamic_version: true
    files:
    - source: dist/pipelines-0.0.1-py3-none-any.whl

resources:
  jobs:
    sync_tables_gold_with_pg:
      name: sync_tables_gold_with_pg

      tasks:
        - task_key: sync_tables_gl_and_pg
          job_cluster_key: job_new_cluster
          existing_cluster_id: 1224-151003-c62b2avz
          notebook_task:
            notebook_path: ../notebook/sync_gold_tables_to_postgres.ipynb
            source: WORKSPACE
            base_parameters:
              env: ${bundle.target}
          libraries:
            - whl: dist/pipelines-0.0.1-py3-none-any.whl





View solution in original post