topic Re: Bug with enabling UniForm Data Format? in Data Engineering

Bug with enabling UniForm Data Format?

stevenayers-bge — Fri, 26 Apr 2024 04:59:41 GMT

In the documentation for enabling iceberg compatibility on delta tables, it states that the minReaderVersion for IcebergCompatV1 and IcebergCompatV2 is 2 (https://docs.databricks.com/en/delta/uniform.html#requirements).

However, when you run the REORG TABLE command to enable uniform format:

REORG TABLE table_name APPLY (UPGRADE UNIFORM(ICEBERG_COMPAT_VERSION=2));

The table's minReaderVersion is upgraded to 3. Is there a reason for this or is this a bug?

Re: Bug with enabling UniForm Data Format?

daniel_sahal — Fri, 26 Apr 2024 06:09:40 GMT

@stevenayers-bge
I've just checked source code of delta and you're right - documentation states that tat minReaderVersion should be >=2, but source code is upgrading it to 3

https://github.com/delta-io/delta/blob/78970abd96dfc0278e21c04cda442bb05ccde4a1/spark/src/main/scala/org/apache/spark/sql/delta/commands/ReorgTableForUpgradeUniformHelper.scala#L80

That's either a bug in a code or mistake in documentation. I'll raise an issue to delta on github.

Re: Bug with enabling UniForm Data Format?

ADB — Fri, 14 Jun 2024 01:18:11 GMT

databricks autoloader is not able to generate iceberg metadata json file for incremental load.For initial Load iceberg medata json file generated.