cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to read Delta Table using external tools

oakhill
New Contributor III

I am using the new credential vending API to get tokens and url for my tables in Unity Catalog.

I get the token, url and I am able to scan the folder using read_parquet, but NOT any Delta Lake functions. Not TableExists, scan_delta or delta_scan from Polars or DuckDB.

The table is written using PySpark, and no particular settings.

When reading from Polars:

 

DeltaError: Table metadata is invalid: Number of checkpoint files '0' is not equal to number of checkpoint metadata parts 'None'

 

When reading from DuckDB:

 

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table:

 

 Is there any Databricks-specifics to reading the deltatables?

When reading the _same folder_ using read_parquet, it works.

3 REPLIES 3

Bernard295Clark
New Contributor II

Hello!

It sounds like you're encountering issues when trying to read Delta Lake tables using Polars and DuckDB, but not with read_parquet. This could be due to Databricks-specific configurations required for Delta Lake tables. Ensure you're using the correct format ("delta") when reading Delta tables. Additionally, verify that the transaction log folder _delta_log is present and correctly configured. 

Yes, everything is working perfectly in Unity Catalog and Databricks. Transaction-folder is present.

M_Pal
New Contributor II

I'm also having the exact same problem as @oakhill , when trying to read from duckdb into any delta tables I get the error. Would it be possible to explain a solution of how the delta extension from duckdb can be working in databricks with Delta Lake, please?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group