cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

dbx execute cluster issue

tariq
New Contributor III

I am facing issue with cluster not being able to run the updated code using dbx execute command. Any changes I make to the code is not reflected in the execution until I restart the cluster. I am using a photon enable cluster with Standard_D4s_v5 as machine type for both driver and worker

4 REPLIES 4

Debayan
Databricks Employee
Databricks Employee

Hi, Could you please elaborate on how "Any changes I make to the code is not reflected in the execution until I restart the cluster"

Also, please tag @Debayan Mukherjeeโ€‹ in your next command so that I will be notified.

tariq
New Contributor III

@Debayan Mukherjeeโ€‹ 

I mean when I execute the following command:

dbx execute --cluster-id=cluster_id job_name

It executes the job/workflow in that cluster and returns any cli output to my local terminal. Now if I make changes/debug to my code and execute again then the new wheel file built is somehow not being use to execute and I get the output from the old code. Only when I restart my cluster, I am able to see any changes in execution. This is a recent behavior and not something I've experienced before and I have been using dbx for some time now.

Debayan
Databricks Employee
Databricks Employee

Thanks for the details.

As suggested in https://docs.databricks.com/dev-tools/dbx.html could you please raise a question in GITHUB so that it can be reviewed by the SMEs?

"This article covers dbx by Databricks Labs, which is provided as-is and is not supported by Databricks through customer technical support channels. Questions and feature requests can be communicated through the Issues page of the databrickslabs/dbx repo on GitHub."

Anonymous
Not applicable

Hi @Tarique Anwarโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group