topic Try catch multiple write streams on a job in Data Engineering

Try catch multiple write streams on a job

AdamRink — Mon, 14 Feb 2022 23:03:40 GMT

We are having issues with checkpoints and schema versions getting out of date (no idea why), but it causes jobs to fail. We have jobs that are running 15-30 streaming queries, so if one fails, that creates an issue. I would like to trap the checkpoint errors, and just reset the checkpoint and log a failure. Not optimal because then I'm reprocessing the stream or at least the window we are looking at.

So the problem I have is it seems the only way to error trap the stream is to use an awaitTermination()... this locks up notebook and the next streams won't start until the first stream is terminated. The awaitAnyTermination() won't catch when the job starts up and an error occurs because the job stops before hitting the awaitAnyTermination?

Re: Try catch multiple write streams on a job

AdamRink — Tue, 15 Feb 2022 19:53:10 GMT

Thx Kaniz

Re: Try catch multiple write streams on a job

AdamRink — Wed, 23 Feb 2022 17:22:41 GMT

The problem is that on startup if a stream fails, it would never hit the awaitAnyTermination? I almost want to take that while loop and put it on a background thread to start that at the beginning and then fire all the streams afterward... not sure if that is possible?

Re: Try catch multiple write streams on a job

jose_gonzalez — Tue, 05 Apr 2022 22:36:52 GMT

Hi @Adam Rink ,

Just a friendly follow-up. Do you still need help or @Kaniz Fatma response help to resolve your question?