dbx execute cluster issue

[tariq]

I am facing issue with cluster not being able to run the updated code using dbx execute command. Any changes I make to the code is not reflected in the execution until I restart the cluster. I am using a photon enable cluster with Standard_D4s_v5 as machine type for both driver and worker

[Debayan]

Hi, Could you please elaborate on how "Any changes I make to the code is not reflected in the execution until I restart the cluster"

Also, please tag @Debayan Mukherjee​ in your next command so that I will be notified.

[tariq]

@Debayan Mukherjee​ 

I mean when I execute the following command:

dbx execute --cluster-id=cluster_id job_name

It executes the job/workflow in that cluster and returns any cli output to my local terminal. Now if I make changes/debug to my code and execute again then the new wheel file built is somehow not being use to execute and I get the output from the old code. Only when I restart my cluster, I am able to see any changes in execution. This is a recent behavior and not something I've experienced before and I have been using dbx for some time now.

[Debayan]

Thanks for the details.

As suggested in https://docs.databricks.com/dev-tools/dbx.html could you please raise a question in GITHUB so that it can be reviewed by the SMEs?

"This article covers dbx by Databricks Labs, which is provided as-is and is not supported by Databricks through customer technical support channels. Questions and feature requests can be communicated through the Issues page of the databrickslabs/dbx repo on GitHub."

[Anonymous]

Hi @Tarique Anwar​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!