How to Retrieve the spark.statistics.createdAt When Statistics Were Last Updated in Databricks?

Sainath368
Contributor

‎08-04-2025 05:20 AM

Hi everyone,

I regularly (once a week) run the analyze table compute statistics command on all my tables in Databricks to keep statistics up to date for query optimization.

In the Spark table UI catalog, I can see some statistics metadata like spark.statistics and spark.statistics.createdAt. However, when I query the table properties or use commands like describe table extended or check the table metadata programmatically, I can only retrieve statistics such as table size and row count — but not the spark.statistics.createdAt value indicating when the statistics were last computed.

Sainath368_0-1754309683688.png

I need to store this updated time in another table . Could anyone suggest a reliable method or the right command to retrieve the exact time when statistics were last updated or created for a table in Databricks/Spark?

Thanks in advance!


0 Kudos

Advika
Community Manager
Community Manager

‎08-05-2025 06:52 AM

Hello @Sainath368!

sql.statistics.createdAt reflects the epoch time when statistics were created. Unfortunately, there's no direct command available to check when the statistics were last updated.

As a workaround, you can manually set the current timestamp as a custom table property right after running the ANALYZE command.

View solution in original post