Improve iteration time on implementing jobs

[jannemanson]

Hey there,

I am using dbx to create Databricks tasks and deploy the job. I find it not ideal since the iteration circles are sometimes a bit long when I have to wait for a job with several tasks to complete and see where it failed.

I am already trying some things to improve my iteration circles such as:

* using pools

* only deploying and running tasks that fail (exclude the others from the deployment)

* testing code snippets in Databricks notebooks or locally

Do you have any additional ideas on how to improve the iteration of jobs? I do not find that quite ideal and a bit cumbersome. I worked once with VCR for playtests and found that very nice. Does there exist something similar for Databricks jobs?

[Debayan]

Hi, you can refer https://docs.databricks.com/clusters/cluster-config-best-practices.html, if this helps.

Please tag @Debayan​ with your next comment so that I will get notified. Thank you!

[Anonymous]

Hi @Jan HE​ 

We haven't heard from you since the last response from @Debayan Mukherjee​ ​, and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

[jannemanson]

Hello, thanks for the answer. Unfortunately, this did not help me, since it is general best practice. @Debayan Mukherjee​