topic Re: Improve iteration time on implementing jobs in Data Engineering

Improve iteration time on implementing jobs

jannemanson — Tue, 02 May 2023 11:09:30 GMT

Hey there,

I am using dbx to create Databricks tasks and deploy the job. I find it not ideal since the iteration circles are sometimes a bit long when I have to wait for a job with several tasks to complete and see where it failed.

I am already trying some things to improve my iteration circles such as:

* using pools

* only deploying and running tasks that fail (exclude the others from the deployment)

* testing code snippets in Databricks notebooks or locally

Do you have any additional ideas on how to improve the iteration of jobs? I do not find that quite ideal and a bit cumbersome. I worked once with VCR for playtests and found that very nice. Does there exist something similar for Databricks jobs?

Re: Improve iteration time on implementing jobs

Anonymous — Fri, 19 May 2023 06:32:34 GMT

Hi @Jan HE

We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: Improve iteration time on implementing jobs

jannemanson — Fri, 19 May 2023 10:14:20 GMT

Hello, thanks for the answer. Unfortunately, this did not help me, since it is general best practice. @Debayan Mukherjee

Re: Improve iteration time on implementing jobs

Debayan — Sun, 07 May 2023 19:24:33 GMT

Hi, you can refer https://docs.databricks.com/clusters/cluster-config-best-practices.html, if this helps.

Please tag @Debayan with your next comment so that I will get notified. Thank you!