I am using MultiThread in this job which creates 8 parallel jobs. It fails for few times in a day and sometimes stuck in any of the Python notebook cell process. Here
The Python process exited with an unknown exit code.
The last 10 KB of the process's stderr and stdout can be found below. See driver logs for full logs.
---------------------------------------------------------------------------
Last messages on stderr:
Tue Feb 7 17:10:18 2023 Connection to spark from PID 24461
Tue Feb 7 17:10:18 2023 Initialized gateway on port 34499
Tue Feb 7 17:10:19 2023 Connected to spark.
---------------------------------------------------------------------------
Last messages on stdout:
NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.
To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.
To read more about this, see https://github.com/ipython/ipython/issues/2049