Thank you for the response. Have you tried remote debugging with Databricks Connect V2? I am particularly interested in stepping into the methods from the libraries installed on the cluster. Is it possible?

As for setting up a local Spark environment with Docker - it is even more complicated in my company due to security restrictions. The setup I mentioned in my post was the best thing I could achieve given the limitations.