Issue importing library deltalake

Gcabrera — Thu, 07 Dec 2023 08:31:08 GMT

Hello,

I'm currently seeing a rather cryptic error message whenever I try to import the deltalake library into Databricks (without actually doing anything else).

import datalake

"ImportError: /local_disk0/.ephemeral_nfs/envs/pythonEnv-cbe496f6-d064-40ae-87c0-ab3d3c23ccb3/lib/python3.9/site-packages/deltalake/_internal.abi3.so: cannot allocate memory in static TLS block"

However the library seems to work just fine locally or within an SH cell.

I have tried quite a few things but nothing really seems to work:

Installing various versions of the library (and its dependencies).
Installing the library via %pip %sh !pip or the cluster/job tab.
Setting a system variable in an SH cell: export LD_PRELOAD
Restarting python interpreter.
Increasing cluster memory.
Only importing a single class from the library.

Any insight you may have would be greatly appreciated.

Best regards,

Gustavo

Re: Issue importing library deltalake

jose_gonzalez — Wed, 10 Jan 2024 19:49:16 GMT

Are you trying to import this library to a Databricks notebook? are you using open source spark in your local machine?

topic Re: Issue importing library deltalake in Data Engineering

Issue importing library deltalake

Re: Issue importing library deltalake