How to track the history of schema changes for a Delta table

[User16869510359]

I have a Delta table that had schema changes in multiple commits. I wanted to track all these schema changes that happened on the Delta table. The "DESCRIBE HISTORY" is not useful as it logs the schema change made by ALTER TABLE operations.

[User16869510359 · Accepted solution]

When a write operation is performed with columns added. we are not explicitly showing that in 

DESCRIBE HISTORY output. Only an entry is made for write. and in the operation Parameters, it's not showing anything about schema evolution. whereas if we do alter table then that is shown explicitly. 

Below code snippet queries the Delta transaction logs to give us the details about all the schema changes done on the table (Replace the table location with your Delta table location)

spark.read.json("dbfs:/tmp/hari/delta/caching/test/2/_delta_log/*.json").createOrReplaceTempView("delta_log")
display(spark.sql("select metadata.schemaString,input_file_name() from delta_log where metadata is not null"))