Notebook fails in job but not in interactive mode

[-werners-]

I have this notebook which is scheduled by Data Factory on a daily basis.

It works fine, up to today. All of a sudden I keep on getting NullpointerException when writing the data.

After some searching online, I disabled AQE. But this does not help.

The weird part is: when I run the notebook manually it does work.

So it seems something has changed in the scheduling of ADF or the job interface of Databricks?

Any idea in which direction I should look?

I also tried using fixed clusters (so no autoscaling), no spot instances, different dbrx-versions (9.0, 9,1, 10.0, 10.1).

[-werners-]

After some tests it seems that if I run the notebook on an interactive cluster, I only get 80% of load (Ganglia metrics).

If I run the same notebook on a job cluster with the same VM types etc (so the only difference is interactive vs job), I get over 150% load => worker crash.

Weird.

[jose_gonzalez]

Did you try on DBR 7.3 LTS? I think there was an issue with NPE that was affecting some versions but not 7.3.

[-werners-]

Hm bizarre, it works fine now with any version.

Must have been some weird data or something.