dbx execute cluster issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ06-14-2023 09:47 AM
I am facing issue with cluster not being able to run the updated code using dbx execute command. Any changes I make to the code is not reflected in the execution until I restart the cluster. I am using a photon enable cluster with Standard_D4s_v5 as machine type for both driver and worker
- Labels:
-
Dbx
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ06-15-2023 12:12 AM
Hi, Could you please elaborate on how "Any changes I make to the code is not reflected in the execution until I restart the cluster"
Also, please tag @Debayan Mukherjeeโ in your next command so that I will be notified.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ06-15-2023 02:24 AM
@Debayan Mukherjeeโ
I mean when I execute the following command:
dbx execute --cluster-id=cluster_id job_name
It executes the job/workflow in that cluster and returns any cli output to my local terminal. Now if I make changes/debug to my code and execute again then the new wheel file built is somehow not being use to execute and I get the output from the old code. Only when I restart my cluster, I am able to see any changes in execution. This is a recent behavior and not something I've experienced before and I have been using dbx for some time now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ06-15-2023 11:39 PM
Thanks for the details.
As suggested in https://docs.databricks.com/dev-tools/dbx.html could you please raise a question in GITHUB so that it can be reviewed by the SMEs?
"This article covers dbx by Databricks Labs, which is provided as-is and is not supported by Databricks through customer technical support channels. Questions and feature requests can be communicated through the Issues page of the databrickslabs/dbx repo on GitHub."
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ06-16-2023 12:34 AM
Hi @Tarique Anwarโ
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!