topic Re: Error Databricks Bundle Deploy with changes in the wheel file in Data Engineering

Error Databricks Bundle Deploy with changes in the wheel file

jeremy98 — Thu, 02 Jan 2025 16:11:21 GMT

Hello Community,
Suddenly, I have an error, when I'm doing the deploy of the new bundle to databricks changing the python script, the cluster continue to point to an old version of the py script uploaded from databricks asset bundle, why this?

Re: Error Databricks Bundle Deploy with changes in the wheel file

jeremy98 — Thu, 02 Jan 2025 16:26:08 GMT

Seems that the problem is to use an existing cluster and making the deploy again and again point to a first upload of the same script.. is this a databricks bug?

Re: Error Databricks Bundle Deploy with changes in the wheel file

Alberto_Umana — Thu, 02 Jan 2025 16:27:53 GMT

Hi @jeremy98,

What is the error that you are hitting?

Re: Error Databricks Bundle Deploy with changes in the wheel file

jeremy98 — Thu, 02 Jan 2025 16:40:28 GMT

Hello Alberto,

Yes, essentially, when I was deploying a workflow based on an existing cluster to run manually inside Databricks, I encountered an issue. While updating a Python script used by a notebook task, the script pointed to the initial version instead of the updated one, even though I redeployed the DAB.

Let me know if this makes things clearer.

Re: Error Databricks Bundle Deploy with changes in the wheel file

Alberto_Umana — Thu, 02 Jan 2025 17:00:15 GMT

Thanks for the comments!

Just to confirm you are following this deployment logic?

databricks bundle validate
databricks bundle deploy -t dev
databricks bundle run -t dev hello_job

Also it looks like existing cluster is not picking up new version you can either restart the cluster manually or automate the restart process as part of your deployment workflow, just to confirm.

Re: Error Databricks Bundle Deploy with changes in the wheel file

jeremy98 — Thu, 02 Jan 2025 23:12:48 GMT

Hello Alberto,
Thanks for your answer, yes basically I did those steps. Sometimes I didn't do validate before deploy (putting -t stg, instead dev target) but also the last command all manually through the portal.

I restarted, but again after 3 deploys for example, the cluster doesn't take the correct version of the py script..

Re: Error Databricks Bundle Deploy with changes in the wheel file

Alberto_Umana — Fri, 03 Jan 2025 12:49:44 GMT

Hi @jeremy98,

Could you please send me your databricks.yml file to review it?

Re: Error Databricks Bundle Deploy with changes in the wheel file

jeremy98 — Mon, 06 Jan 2025 19:04:48 GMT

Hello,
This is the yaml file I used, but btw I think because I cannot use a cluster up and never stop, I had the problem that during every new deploy, btw the cluster retains the first deploy... seems weird.

resources: jobs: sync_tables_gold_with_pg: name: sync_tables_gold_with_pg tasks: - task_key: sync_tables_gl_and_pg job_cluster_key: job_new_cluster existing_cluster_id: 1224-151003-c62b2avz notebook_task: notebook_path: ../notebook/sync_gold_tables_to_postgres.ipynb source: WORKSPACE base_parameters: env: ${bundle.target} libraries: - whl: ${workspace.root_path}/files/dist/pipelines-0.0.1-py3-none-any.whl

Re: Error Databricks Bundle Deploy with changes in the wheel file

denis-dbx — Wed, 09 Apr 2025 09:35:54 GMT

We've added a solution for this problem in v0.245.0. There is opt-in "dynamic_version: true" flag on artifact to enable automated wheel patching that break the cache (Example). Once set, "bundle deploy" will transparently patch version suffix in the built wheel so that the new code is always deployed.

Note, that this only works with local wheels that are uploaded as part of "bundle deploy". So you need to download this wheel into your project so that it can be patched by "bundle deploy" before upload to break the cache.

artifacts: mywheel: type: whl dynamic_version: true files: - source: dist/pipelines-0.0.1-py3-none-any.whl resources: jobs: sync_tables_gold_with_pg: name: sync_tables_gold_with_pg tasks: - task_key: sync_tables_gl_and_pg job_cluster_key: job_new_cluster existing_cluster_id: 1224-151003-c62b2avz notebook_task: notebook_path: ../notebook/sync_gold_tables_to_postgres.ipynb source: WORKSPACE base_parameters: env: ${bundle.target} libraries: - whl: dist/pipelines-0.0.1-py3-none-any.whl

Re: Error Databricks Bundle Deploy with changes in the wheel file

jeremy98 — Wed, 09 Apr 2025 16:06:05 GMT

AMAZING :), thx denis!