05-02-2023 04:09 AM
Hey there,
I am using dbx to create Databricks tasks and deploy the job. I find it not ideal since the iteration circles are sometimes a bit long when I have to wait for a job with several tasks to complete and see where it failed.
I am already trying some things to improve my iteration circles such as:
* using pools
* only deploying and running tasks that fail (exclude the others from the deployment)
* testing code snippets in Databricks notebooks or locally
Do you have any additional ideas on how to improve the iteration of jobs? I do not find that quite ideal and a bit cumbersome. I worked once with VCR for playtests and found that very nice. Does there exist something similar for Databricks jobs?
05-07-2023 12:24 PM
Hi, you can refer https://docs.databricks.com/clusters/cluster-config-best-practices.html, if this helps.
Please tag @Debayan with your next comment so that I will get notified. Thank you!
05-18-2023 11:32 PM
Hi @Jan HE
We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
05-19-2023 03:14 AM
Hello, thanks for the answer. Unfortunately, this did not help me, since it is general best practice. @Debayan Mukherjee
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.