โ01-02-2025 08:11 AM
Hello Community,
Suddenly, I have an error, when I'm doing the deploy of the new bundle to databricks changing the python script, the cluster continue to point to an old version of the py script uploaded from databricks asset bundle, why this?
โ01-02-2025 08:26 AM
Seems that the problem is to use an existing cluster and making the deploy again and again point to a first upload of the same script.. is this a databricks bug?
โ01-02-2025 08:27 AM
Hi @jeremy98,
What is the error that you are hitting?
โ01-02-2025 08:40 AM
Hello Alberto,
Yes, essentially, when I was deploying a workflow based on an existing cluster to run manually inside Databricks, I encountered an issue. While updating a Python script used by a notebook task, the script pointed to the initial version instead of the updated one, even though I redeployed the DAB.
Let me know if this makes things clearer.
โ01-02-2025 09:00 AM
Thanks for the comments!
Just to confirm you are following this deployment logic?
databricks bundle validate
databricks bundle deploy -t dev
databricks bundle run -t dev hello_job
Also it looks like existing cluster is not picking up new version you can either restart the cluster manually or automate the restart process as part of your deployment workflow, just to confirm.
โ01-02-2025 03:12 PM
Hello Alberto,
Thanks for your answer, yes basically I did those steps. Sometimes I didn't do validate before deploy (putting -t stg, instead dev target) but also the last command all manually through the portal.
I restarted, but again after 3 deploys for example, the cluster doesn't take the correct version of the py script..
โ01-03-2025 04:49 AM
Hi @jeremy98,
Could you please send me your databricks.yml file to review it?
โ01-06-2025 11:02 AM - edited โ01-06-2025 11:04 AM
Hello,
This is the yaml file I used, but btw I think because I cannot use a cluster up and never stop, I had the problem that during every new deploy, btw the cluster retains the first deploy... seems weird.
resources:
jobs:
sync_tables_gold_with_pg:
name: sync_tables_gold_with_pg
tasks:
- task_key: sync_tables_gl_and_pg
job_cluster_key: job_new_cluster
existing_cluster_id: 1224-151003-c62b2avz
notebook_task:
notebook_path: ../notebook/sync_gold_tables_to_postgres.ipynb
source: WORKSPACE
base_parameters:
env: ${bundle.target}
libraries:
- whl: ${workspace.root_path}/files/dist/pipelines-0.0.1-py3-none-any.whl
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group