Databricks notebook taking too long to run as a job compared to when triggered from within the notebook

[curious-case-of]

I don't know if this question has been covered earlier, but here it goes - I have a notebook that I can run manually using the 'Run' button in the notebook or as a job.

The runtime when I run from within the notebook directly is roughly 2 hours. But when I execute it as a job, the runtime is huge (around 8 hours)

. The piece of code which takes the longest time is calling an applyInPandas function, which in turn calls a pandas_udf which trains an auto_arima model (pmdarima).

Can anyone help me figure out what might be happening? I am clueless.

Thanks!

[Kaniz · Accepted solution]

Hi @Vidisha Kanodia​ ,

There is a similar thread - https://community.databricks.com/s/question/0D53f00001pCk29CAC/performance-for-pyspark-dataframe-is-...

Please have a look at that.

[Kaniz]

Hi @Vidisha Kanodia​ , Just a friendly follow-up. Do you still need help? Please let us know.

[wvl]

We're seeing the same behavior.. Good performance using interactive cluster.

Using identically sized job cluster, performance is bad.

Any ideas?