topic Re: dbx execute cluster issue in Machine Learning

dbx execute cluster issue

tariq — Wed, 14 Jun 2023 16:47:50 GMT

I am facing issue with cluster not being able to run the updated code using dbx execute command. Any changes I make to the code is not reflected in the execution until I restart the cluster. I am using a photon enable cluster with Standard_D4s_v5 as machine type for both driver and worker

Re: dbx execute cluster issue

tariq — Thu, 15 Jun 2023 09:24:48 GMT

@Debayan Mukherjee

I mean when I execute the following command:

dbx execute --cluster-id=cluster_id job_name

It executes the job/workflow in that cluster and returns any cli output to my local terminal. Now if I make changes/debug to my code and execute again then the new wheel file built is somehow not being use to execute and I get the output from the old code. Only when I restart my cluster, I am able to see any changes in execution. This is a recent behavior and not something I've experienced before and I have been using dbx for some time now.

Re: dbx execute cluster issue

Anonymous — Fri, 16 Jun 2023 07:34:08 GMT

Hi @Tarique Anwar

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: dbx execute cluster issue

Debayan — Thu, 15 Jun 2023 07:12:13 GMT

Hi, Could you please elaborate on how "Any changes I make to the code is not reflected in the execution until I restart the cluster"

Also, please tag @Debayan Mukherjee in your next command so that I will be notified.

Re: dbx execute cluster issue

Debayan — Fri, 16 Jun 2023 06:39:25 GMT

Thanks for the details.

As suggested in https://docs.databricks.com/dev-tools/dbx.html could you please raise a question in GITHUB so that it can be reviewed by the SMEs?

"This article covers dbx by Databricks Labs, which is provided as-is and is not supported by Databricks through customer technical support channels. Questions and feature requests can be communicated through the Issues page of the databrickslabs/dbx repo on GitHub."