Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

[gayatrikhatale]

Hi, 

I had a query regarding how to accurately retrieve the last data refresh datetime and last metadata refresh datetime for tables in Databricks. Currently, the only reliable approach I am aware of is using the DESCRIBE HISTORY command with filters on operations column such as WRITE, UPDATE, DELETE, MERGE, or ALTER TABLE.

However, I could not find a comprehensive list of all possible operations that can appear in the operation column of DESCRIBE HISTORY so I can separate them. The documentation here — Databricks Delta Table History — only provides examples of operation metric key values, but not the full list of supported operations.

Could you please help me.

Thank you!

[szymon_dybczak]

Hi @gayatrikhatale,

I think you already found correct place in documentation. It lists all supported operations in the first column.

https://docs.databricks.com/aws/en/delta/history?#operation-metrics-key

[gayatrikhatale]

Hi @szymon_dybczak ,

Thank you for your response.

If this is the correct documentation, then why am I not able to see operations like ADD COLUMNS or DROP COLUMNS listed there? I have observed these operation types in the output of DESCRIBE HISTORY, but they don’t seem to be documented in the link.

Thank you!

[szymon_dybczak]

Thanks for information. To be honest I've always thought that this table contains all supported operations. I learn everyday something new 🙂

[siva-anantha]

Hi, 

I had a similar need once and referred the delta-io git repository for full list of metrics. It is the not the best way but we get an idea of what all is available in the metrics. Hope it helps.

Regards,
Siva

[szymon_dybczak]

Hi @siva-anantha ,

This is super useful, thanks for sharing. I've prepared list of all distinct operation based on source code for reference:

• WRITE
• STREAMING UPDATE
• DELETE
• UPDATE
• MERGE
• TRUNCATE
• CREATE TABLE
• REPLACE TABLE
• ADD COLUMNS
• DROP COLUMNS
• RENAME COLUMN
• CHANGE COLUMN
• CHANGE COLUMNS
• REPLACE COLUMNS
• UPDATE SCHEMA
• UPDATE COLUMN METADATA
• SET TBLPROPERTIES
• UNSET TBLPROPERTIES
• UPGRADE PROTOCOL
• DROP FEATURE
• REMOVE COLUMN MAPPING
• OPTIMIZE
• REORG
• VACUUM START
• VACUUM END
• COMPUTE STATS
• RESTORE
• CLONE
• CONVERT
• ADD CONSTRAINT
• DROP CONSTRAINT
• CLUSTER BY
• UPGRADE UNIFORM PROPERTIES
• ROW TRACKING BACKFILL
• ROW TRACKING UNBACKFILL
• Manual Update
• Empty Commit
• Deletion Vector Tombstones
• Domain Metadata Cleanup

[gayatrikhatale]

Thank you @szymon_dybczak , @siva-anantha !