topic Re: Unable to read Delta Table using external tools in Data Engineering

Unable to read Delta Table using external tools

oakhill — Tue, 28 Jan 2025 09:32:17 GMT

I am using the new credential vending API to get tokens and url for my tables in Unity Catalog.

I get the token, url and I am able to scan the folder using read_parquet, but NOT any Delta Lake functions. Not TableExists, scan_delta or delta_scan from Polars or DuckDB.

The table is written using PySpark, and no particular settings.

When reading from Polars:

DeltaError: Table metadata is invalid: Number of checkpoint files '0' is not equal to number of checkpoint metadata parts 'None'

When reading from DuckDB:

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table:

Is there any Databricks-specifics to reading the deltatables?

When reading the _same folder_ using read_parquet, it works.

Re: Unable to read Delta Table using external tools

Bernard295Clark — Tue, 28 Jan 2025 11:55:30 GMT

Hello!

It sounds like you're encountering issues when trying to read Delta Lake tables using Polars and DuckDB, but not with read_parquet. This could be due to Databricks-specific configurations required for Delta Lake tables. Ensure you're using the correct format ("delta") when reading Delta tables. Additionally, verify that the transaction log folder _delta_log is present and correctly configured.

Re: Unable to read Delta Table using external tools

oakhill — Wed, 29 Jan 2025 15:05:36 GMT

Yes, everything is working perfectly in Unity Catalog and Databricks. Transaction-folder is present.

Re: Unable to read Delta Table using external tools

M_Pal — Mon, 17 Feb 2025 16:01:54 GMT

I'm also having the exact same problem as @oakhill , when trying to read from duckdb into any delta tables I get the error. Would it be possible to explain a solution of how the delta extension from duckdb can be working in databricks with Delta Lake, please?

Re: Unable to read Delta Table using external tools

matovitch — Tue, 22 Apr 2025 13:42:53 GMT

When copying a problematic delta table and reading the copy the issue disappear, it seems to be related to the new delta checkpointPolicy (v2) not supported by the rust implementation of delta but fine with the scala/java one (deltalake vs delta-spark).

https://github.com/delta-io/delta-rs/issues/3215