topic Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks in Data Engineering

Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

gayatrikhatale — Thu, 18 Sep 2025 17:22:35 GMT

Hi,

I had a query regarding how to accurately retrieve the last data refresh datetime and last metadata refresh datetime for tables in Databricks. Currently, the only reliable approach I am aware of is using the DESCRIBE HISTORY command with filters on operations column such as WRITE, UPDATE, DELETE, MERGE, or ALTER TABLE.

However, I could not find a comprehensive list of all possible operations that can appear in the operation column of DESCRIBE HISTORY so I can separate them. The documentation here — Databricks Delta Table History — only provides examples of operation metric key values, but not the full list of supported operations.

Could you please help me.

Thank you!

Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

szymon_dybczak — Thu, 18 Sep 2025 17:46:24 GMT

Hi @gayatrikhatale,

I think you already found correct place in documentation. It lists all supported operations in the first column.

https://docs.databricks.com/aws/en/delta/history?#operation-metrics-key

Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

gayatrikhatale — Thu, 18 Sep 2025 17:53:09 GMT

Hi @szymon_dybczak ,

Thank you for your response.

If this is the correct documentation, then why am I not able to see operations like ADD COLUMNS or DROP COLUMNS listed there? I have observed these operation types in the output of DESCRIBE HISTORY, but they don’t seem to be documented in the link.

Thank you!

Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

siva-anantha — Thu, 18 Sep 2025 18:08:33 GMT

Hi,

I had a similar need once and referred the delta-io git repository for full list of metrics. It is the not the best way but we get an idea of what all is available in the metrics. Hope it helps.

Regards,
Siva

Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

szymon_dybczak — Thu, 18 Sep 2025 19:13:35 GMT

Thanks for information. To be honest I've always thought that this table contains all supported operations. I learn everyday something new 🙂

Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

szymon_dybczak — Thu, 18 Sep 2025 20:22:10 GMT

Hi @siva-anantha ,

This is super useful, thanks for sharing. I've prepared list of all distinct operation based on source code for reference:

WRITE
STREAMING UPDATE
DELETE
UPDATE
MERGE
TRUNCATE
CREATE TABLE
REPLACE TABLE
ADD COLUMNS
DROP COLUMNS
RENAME COLUMN
CHANGE COLUMN
CHANGE COLUMNS
REPLACE COLUMNS
UPDATE SCHEMA
UPDATE COLUMN METADATA
SET TBLPROPERTIES
UNSET TBLPROPERTIES
UPGRADE PROTOCOL
DROP FEATURE
REMOVE COLUMN MAPPING
OPTIMIZE
REORG
VACUUM START
VACUUM END
COMPUTE STATS
RESTORE
CLONE
CONVERT
ADD CONSTRAINT
DROP CONSTRAINT
CLUSTER BY
UPGRADE UNIFORM PROPERTIES
ROW TRACKING BACKFILL
ROW TRACKING UNBACKFILL
Manual Update
Empty Commit
Deletion Vector Tombstones
Domain Metadata Cleanup

Re: Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks

gayatrikhatale — Fri, 19 Sep 2025 09:44:28 GMT

Thank you @szymon_dybczak , @siva-anantha !