cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta table version protocol

noname123
New Contributor III

I do:

df.write.format("delta").mode("append").partitionBy("timestamp").option("mergeSchema", "true").save(destination)

If table doesn't exist, it creates new table with "minReaderVersion":3,"minWriterVersion":7.
Yesterday it was creating table with "minReaderVersion":1,"minWriterVersion":2. I changed nothing and now versions are different. Setting delta protocol version on cluster does nothing.
This causes breaking change to me because some tables are used by Azure Data Factory and it's not compatible.
1 ACCEPTED SOLUTION

Accepted Solutions

noname123
New Contributor III

Thanks for help.

Issue was caused by "Auto-Enable Deletion Vectors" setting. 

View solution in original post

2 REPLIES 2

Kaniz
Community Manager
Community Manager

Hi @noname123, I understand the issue you’re facing with Delta Lake protocol versions.

Let’s break it down:

  1. Delta Lake Protocol Versions:

  2. Your Scenario:

    • You’ve noticed that the protocol versions have changed unexpectedly.
    • Previously, the table was created with “minReaderVersion”:1 and “minWriterVersion”:2.
    • Now, it’s creating tables with “minReaderVersion”:3 and “minWriterVersion”:7.
    • This discrepancy is causing compatibility issues with Azure Data Factory.
  3. Options and Considerations:

  4. Next Steps:

    • Evaluate the impact of the protocol version change on your existing workflows.
    • If possible, coordinate with other users or teams to ensure compatibility.
    • Consider upgrading Azure Data Factory to a version that supports the new protocol version.
    • If necessary, explore alternative solutions or workarounds based on your specific requirements.

Remember that protocol version changes should be approached carefully, especially in production environments. It’s essential to balance the benefits of new features with potential disruptions to existing processes. 🛠

 

noname123
New Contributor III

Thanks for help.

Issue was caused by "Auto-Enable Deletion Vectors" setting. 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!