For stability, I've stuck with LTS. Last Friday my containers stopped working with error message:

Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 2442, in _call_proxy
    return_value = getattr(self.pool[obj_id], method)(*params)
  File "/databricks/python_shell/dbruntime/pythonPathHook.py", line 45, in initStartingDirectory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/Git-repos/'

After spending hours and hours of troubleshooting and nothing working, I went for runtime 10.2 (the non-LTS -version), which does not have this problem.

It looks like a regression problem to me. JRE got updated for sure (Log4J-fixes) and this inability to run containers wasn't properly covered with automated tests.

Any ideas how to get past the error? As this error happens before Python, there is very little I can do to fix this in my container.