topic Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix? in Data Engineering

Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Mon, 03 Jan 2022 07:46:06 GMT

For stability, I've stuck with LTS. Last Friday my containers stopped working with error message:

Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 2442, in _call_proxy
    return_value = getattr(self.pool[obj_id], method)(*params)
  File "/databricks/python_shell/dbruntime/pythonPathHook.py", line 45, in initStartingDirectory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Repos/Git-repos/'

After spending hours and hours of troubleshooting and nothing working, I went for runtime 10.2 (the non-LTS -version), which does not have this problem.

It looks like a regression problem to me. JRE got updated for sure (Log4J-fixes) and this inability to run containers wasn't properly covered with automated tests.

Any ideas how to get past the error? As this error happens before Python, there is very little I can do to fix this in my container.

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

Hubert-Dudek — Mon, 03 Jan 2022 11:00:44 GMT

First time I see that error but you can try disable/enable repos in admin settings in your databricks workspace

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Mon, 03 Jan 2022 11:05:36 GMT

If I disable repos, from where do my notebooks come from?

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

Hubert-Dudek — Mon, 03 Jan 2022 11:12:31 GMT

I thought just for test to check is it help. Maybe re-enabling it will recover structure.

Additionally that error looks like git is disabled...

(notebooks come from workspace or git repo which is part of workspace)

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Mon, 03 Jan 2022 12:13:19 GMT

On a second thought, your idea is something I might try. However, by using a newer bricks node -version the problem dissolves. Mind you, I've been using LTS-node since May 2021 without problems.

All this leaves me conflicted. Should I reconnect the git-repo and see if that would help or ignore the fact LTS-version is unusable with Python-containers and keep my current compute-configuration.

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

Anonymous — Fri, 14 Jan 2022 17:51:39 GMT

@Jari Turkia - Checking in to see how it went.

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Wed, 19 Jan 2022 08:12:42 GMT

This is getting worse. Now JDBC write to SQL is failing for same reason. I haven't yet found a solution for this.

Am I not supposed to use containers? Python?

This is not cool. 😞

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

Atanu — Wed, 26 Jan 2022 04:38:17 GMT

We need detail log what's is happening inside to troubleshoot this. Do you have our support access?

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Wed, 26 Jan 2022 07:33:59 GMT

No. All I have is access to Azure with Bricks in it.

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Thu, 03 Mar 2022 10:05:39 GMT

Long time has passed and I have a solution to get things working. I do have the logs, but haven't really paid much attention to this lately.

To me, this is a regression issue in Azure Databricks. When a new version of a node is released, it should be tested with containers too. If old node versions are not, maybe I should be informed about it.

Re: Azure Databricks container runtime broken in 9.1 LTS, how to fix?

HQJaTu — Thu, 03 Mar 2022 12:45:56 GMT

Latest node version seems to work for me. Choosing different containers wasn't a factor, the problem was with LXC. I haven't done any follow-up on this, also my containers are custom, see my pull request https://github.com/databricks/containers/pull/73