import error with typing_extensions, issue with pyiceberg and pydantic

xecel — Sat, 31 Aug 2024 01:40:46 GMT

Hello All,

I am currently working in a Databricks environment where I am trying to use the `pyiceberg` library to interact with Iceberg table metadata directly in Unity catalog enabled. However, I'm encountering an issue with package compatibility related to the `typing_extensions` module. Specifically, when I try to import `StaticTable` from `pyiceberg`, I receive the following error: ImportError: cannot import name 'deprecated' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)

Databricks Runtime Version: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)
Installed Packages: pyiceberg, pydantic, typing_extensions

Attempted to run the following code:

from pyiceberg.table import StaticTable

# Replaced Iceberg metadata path
iceberg_table = StaticTable.from_metadata(location)
data = iceberg_table.scan().to_pandas()

I would appreciate any guidance on how to resolve this compatibility issue, or if there is an alternative method for interacting with Iceberg metadata within Databricks that avoids this error.

Re: import error with typing_extensions, issue with pyiceberg and pydantic

Brahmareddy — Mon, 02 Sep 2024 03:23:31 GMT

Hi @xecel, How are you doing today?

As per my understanding, Ensure you're using a compatible version of typing_extensions by installing a specific version like 4.4.0 that might work with pyiceberg. Try reinstalling the libraries (pyiceberg and typing_extensions ) to ensure compatibility. Check the release notes or GitHub issues for pyiceberg to see if there are any updates or known issues with your Databricks runtime version. If the problem persists, consider using Databricks APIs or Spark SQL to interact with Iceberg tables as an alternative. Lastly, think about setting up a custom Python environment to better control library versions.

Just give a try and let me know if it works.

Regards,

Brahma

topic import error with typing_extensions, issue with pyiceberg and pydantic in Data Engineering

import error with typing_extensions, issue with pyiceberg and pydantic

Re: import error with typing_extensions, issue with pyiceberg and pydantic