topic Notebook fails in job but not in interactive mode in Data Engineering

Notebook fails in job but not in interactive mode

-werners- — Thu, 09 Dec 2021 11:27:53 GMT

I have this notebook which is scheduled by Data Factory on a daily basis.

It works fine, up to today. All of a sudden I keep on getting NullpointerException when writing the data.

After some searching online, I disabled AQE. But this does not help.

The weird part is: when I run the notebook manually it does work.

So it seems something has changed in the scheduling of ADF or the job interface of Databricks?

Any idea in which direction I should look?

I also tried using fixed clusters (so no autoscaling), no spot instances, different dbrx-versions (9.0, 9,1, 10.0, 10.1).

Re: Notebook fails in job but not in interactive mode

-werners- — Thu, 09 Dec 2021 15:53:30 GMT

After some tests it seems that if I run the notebook on an interactive cluster, I only get 80% of load (Ganglia metrics).

If I run the same notebook on a job cluster with the same VM types etc (so the only difference is interactive vs job), I get over 150% load => worker crash.

Weird.

Re: Notebook fails in job but not in interactive mode

jose_gonzalez — Fri, 10 Dec 2021 23:11:07 GMT

Did you try on DBR 7.3 LTS? I think there was an issue with NPE that was affecting some versions but not 7.3.

Re: Notebook fails in job but not in interactive mode

-werners- — Mon, 13 Dec 2021 07:52:54 GMT

Hm bizarre, it works fine now with any version.

Must have been some weird data or something.