How to track the history of schema changes for a Delta table

brickster_2018 — Thu, 24 Jun 2021 09:12:23 GMT

I have a Delta table that had schema changes in multiple commits. I wanted to track all these schema changes that happened on the Delta table. The "DESCRIBE HISTORY" is not useful as it logs the schema change made by ALTER TABLE operations.

Re: How to track the history of schema changes for a Delta table

brickster_2018 — Thu, 24 Jun 2021 09:14:44 GMT

When a write operation is performed with columns added. we are not explicitly showing that in

DESCRIBE HISTORY output. Only an entry is made for write. and in the operation Parameters, it's not showing anything about schema evolution. whereas if we do alter table then that is shown explicitly.

Below code snippet queries the Delta transaction logs to give us the details about all the schema changes done on the table (Replace the table location with your Delta table location)

spark.read.json("dbfs:/tmp/hari/delta/caching/test/2/_delta_log/*.json").createOrReplaceTempView("delta_log")
display(spark.sql("select metadata.schemaString,input_file_name() from delta_log where metadata is not null"))

topic Re: How to track the history of schema changes for a Delta table in Data Engineering

How to track the history of schema changes for a Delta table

Re: How to track the history of schema changes for a Delta table