topic Re: Runtime 11.3 LTS not working in my production in Data Engineering

Runtime 11.3 LTS not working in my production

biafch — Thu, 29 Aug 2024 13:31:20 GMT

Hello,

I have a cluster with Runtime 11.3 LTS in my production. Whenever I start this up and try to run my notebooks it's giving me error: Failure starting repl. Try detaching and re-attaching the notebook.

I have a cluster with the same Runtime in my DEV environment and here it's working perfectly fine.

When I downgrade the runtime of my cluster to 10.4 LTS in my production it's working as well.

What can I do to solve this?

Re: Runtime 11.3 LTS not working in my production

biafch — Fri, 30 Aug 2024 06:18:35 GMT

Bump

Re: Runtime 11.3 LTS not working in my production

biafch — Fri, 30 Aug 2024 07:20:28 GMT

Just in case anyone needs to know how to solve this in the future.

Apparently one of my clusters was suddenly having library compatibility issues. Mainly between pandas,numpy and pyarrow.

So I fixed this by forcing specific versions in my global init script. I did the following:

/databricks/python/bin/pip install pandas==2.2.2
/databricks/python/bin/pip install numpy==1.26.4
/databricks/python/bin/pip install pyarrow==7.0.0

And this has solved the problem for me.