Hi Databricks Support Team,
We are raising this ticket to report a cross-engine compatibility issue between Databricks and Teradata when performing DELETE operations on an Iceberg V3 table.
---
ENVIRONMENT DETAILS
-------------------
Databricks Runtime Version : 17.3 LTS (Apache Spark 4.0.0, Scala 2.13)
Iceberg Table Format : V3
External Engine : Teradata (OTF — Open Table Format reader)
Storage : AWS S3
Catalog : Iceberg REST Catalog (Unity Catalog)
PROBLEM DESCRIPTION
-------------------
We have an Iceberg V3 table created from Teradata with the following properties:
'format-version' = '3'
'write.delete.mode' = 'merge-on-read'
'write.update.mode' = 'merge-on-read'
When DELETE operations are performed from Databricks on this table, Databricks generates deletion vectors as .bin files in the S3 location instead of Puffin format (.puffin) as defined in the Iceberg V3 specification.
Subsequent SELECT queries from Teradata OTF fail with the following error:
"Failure 6325 OTF error: Failed to read Puffin file: Failed to fetch manifest file object"
Teradata's OTF reader supports only the Puffin file format for deletion vectors as per the Iceberg V3 spec. It cannot read the .bin deletion vector format that Databricks generates.
---
STEPS TO REPRODUCE
------------------
1. Create Iceberg V3 table from Teradata with merge-on-read delete mode
2. Insert data from Teradata
3. Perform DELETE from Databricks:
DELETE FROM table WHERE mod(i1, 2) = 0;
DELETE FROM table WHERE i1 BETWEEN 5 AND 10;
4. Run SELECT from Teradata — fails with error 6325
5. Check S3 location — only .bin files generated, no .puffin files
---
QUESTIONS
---------
1. Is there a configuration or table property in Databricks DBR 17.3 to force deletion vectors to be written in Puffin format instead of .bin format for Iceberg V3 tables?
2. Is this a known limitation for cross-engine Iceberg V3 interoperability?
3. Is there a planned fix or roadmap item to support Puffin-format deletion vectors in Databricks for Iceberg V3 tables?
