cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

import error with typing_extensions, issue with pyiceberg and pydantic

xecel
New Contributor

Hello All,

I am currently working in a Databricks environment where I am trying to use the `pyiceberg` library to interact with Iceberg table metadata directly in Unity catalog enabled. However, I'm encountering an issue with package compatibility related to the `typing_extensions` module. Specifically, when I try to import `StaticTable` from `pyiceberg`, I receive the following error: ImportError: cannot import name 'deprecated' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)

Databricks Runtime Version: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)
Installed Packages: pyiceberg, pydantic, typing_extensions

Attempted to run the following code:

from pyiceberg.table import StaticTable

# Replaced Iceberg metadata path
iceberg_table = StaticTable.from_metadata(location)
data = iceberg_table.scan().to_pandas()

I would appreciate any guidance on how to resolve this compatibility issue, or if there is an alternative method for interacting with Iceberg metadata within Databricks that avoids this error.

 

1 REPLY 1

Brahmareddy
Valued Contributor II

Hi @xecel, How are you doing today?

As per my understanding, Ensure you're using a compatible version of typing_extensions by installing a specific version like 4.4.0 that might work with pyiceberg. Try reinstalling the libraries (pyiceberg and typing_extensions ) to ensure compatibility. Check the release notes or GitHub issues for pyiceberg to see if there are any updates or known issues with your Databricks runtime version. If the problem persists, consider using Databricks APIs or Spark SQL to interact with Iceberg tables as an alternative. Lastly, think about setting up a custom Python environment to better control library versions.

Just give a try and let me know if it works.

Regards,

Brahma

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group