Here are some quick workarounds that you can try

1. Development mode keeps a cluster warm for rapid iteration. Production mode stops the cluster right after the run finishes. If you must stay in dev mode, tune the pipelines.clusterShutdown.delay so the cluster doesn’t linger. Change the mode for cost savings.

2. In the driver logs, you’ll see the NumberFormatException repeating roughly every minute even after the pipeline reports “completed”. That’s the smoking gun.  If you’re on a recent DBR (e.g., 15.x/16.x), try pinning the pipeline to DBR 14.3 LTS or, conversely, to the latest LTS to see if the ADLS client code path differs.