Unable to read Delta Table using external tools

[oakhill]

I am using the new credential vending API to get tokens and url for my tables in Unity Catalog.

I get the token, url and I am able to scan the folder using read_parquet, but NOT any Delta Lake functions. Not TableExists, scan_delta or delta_scan from Polars or DuckDB.

The table is written using PySpark, and no particular settings.

When reading from Polars:

DeltaError: Table metadata is invalid: Number of checkpoint files '0' is not equal to number of checkpoint metadata parts 'None'

When reading from DuckDB:

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table:

 Is there any Databricks-specifics to reading the deltatables?

When reading the _same folder_ using read_parquet, it works.

[Bernard295Clark]

Hello!

It sounds like you're encountering issues when trying to read Delta Lake tables using Polars and DuckDB, but not with read_parquet. This could be due to Databricks-specific configurations required for Delta Lake tables. Ensure you're using the correct format ("delta") when reading Delta tables. Additionally, verify that the transaction log folder _delta_log is present and correctly configured. 

[oakhill]

Yes, everything is working perfectly in Unity Catalog and Databricks. Transaction-folder is present.

[M_Pal]

I'm also having the exact same problem as @oakhill , when trying to read from duckdb into any delta tables I get the error. Would it be possible to explain a solution of how the delta extension from duckdb can be working in databricks with Delta Lake, please?