Queries from Serverless compute referring to older/deleted/vacuumed version of the delta tables.

s_agarwal
New Contributor

Hi Team,

I have a unity catalog based managed delta table which I am able to successfully query using the regular compute/cluster options.

But when I try to query the same table using a Serverless/SQL Warehouse, they are referring to an older version / metadata of the table which does not exists and I get an error stating that the path does not exists.

In my databricks UI, in the details section of the same table, I see that the property "lastUpdateVersion" points to the exact version which Serverless is trying to read. But my other (non-serverless) computes are reading correct data.

Any help/insight would be appreciated.

Saritha_S
Databricks Employee
Databricks Employee

Hi @s_agarwal 

Please find below my findinsg for your query. 

  • Serverless uses cached Unity Catalog metadata

  • Your UC metadata points to an old Delta version

  • Regular clusters bypass this cache

  • Fix by refreshing or forcing UC metadata rewrite