Delta table version protocol

[noname123]

I do:

df.write.format("delta").mode("append").partitionBy("timestamp").option("mergeSchema", "true").save(destination)

If table doesn't exist, it creates new table with "minReaderVersion":3,"minWriterVersion":7.

Yesterday it was creating table with "minReaderVersion":1,"minWriterVersion":2. I changed nothing and now versions are different. Setting delta protocol version on cluster does nothing.

This causes breaking change to me because some tables are used by Azure Data Factory and it's not compatible.

[Kaniz]

Hi @noname123, I understand the issue you’re facing with Delta Lake protocol versions.

Let’s break it down:

1. Delta Lake Protocol Versions:

   • Delta Lake uses protocol versions to manage compatibility between readers and writers.
   • Protocol version upgrades are irreversible and can impact existing readers and writers.
   • Upgrading the protocol version may break existing Delta Lake table readers, writers, or both.
   • Therefore, it’s essential to upgrade specific tables only when necessary, such as to opt-in to new f...¹.

2. Your Scenario:

   • You’ve noticed that the protocol versions have changed unexpectedly.
   • Previously, the table was created with “minReaderVersion”:1 and “minWriterVersion”:2.
   • Now, it’s creating tables with “minReaderVersion”:3 and “minWriterVersion”:7.
   • This discrepancy is causing compatibility issues with Azure Data Factory.

3. Options and Considerations:

   • Unfortunately, you cannot downgrade the protocol version. It’s an irreversible operation².
   • Dropping and recreating the Delta table with the desired protocol version might be an option, but it depends on your use case and data retention requirements.
   • Ensure that your current and future production tools (like Azure Data Factory) support Delta Lake ta...².

4. Next Steps:

   • Evaluate the impact of the protocol version change on your existing workflows.
   • If possible, coordinate with other users or teams to ensure compatibility.
   • Consider upgrading Azure Data Factory to a version that supports the new protocol version.
   • If necessary, explore alternative solutions or workarounds based on your specific requirements.

Remember that protocol version changes should be approached carefully, especially in production environments. It’s essential to balance the benefits of new features with potential disruptions to existing processes. 🛠️

[noname123 · Accepted solution]

Thanks for help.

Issue was caused by "Auto-Enable Deletion Vectors" setting.