Databricks Community

Smriti2 · a week ago

I recently ran into an interesting Databricks behavior while implementing a row-count comparison using Delta Time Travel (VERSION AS OF).

Platform: Azure
Scenario:

Same Unity Catalog
Same fully qualified table
Same table ID, location, and Delta format

Yet the behavior differed across environments.

What worked in Dev

I ran the notebook interactively
Using an all-purpose cluster
Delta Time Travel (VERSION AS OF) worked as expected

What failed in Prod

The same notebook ran as a scheduled Job
Executed on a job cluster on prod workspace with scheduled job that has one task with a notebook
The exact same Delta table failed with:

TIME TRAVEL is not allowed. Operation not supported on Streaming Tables

The surprising part

The table itself was unchanged:

Same catalog
Same location
Same Delta properties
Same table ID

My code compares active row counts between the last two Delta versions of a table, and fails if the row count drops more than 15%, using Delta time travel (VERSION AS OF) to read past snapshots.

pradeep_singh · a week ago

This could happen if the Delta table in production is a streaming table, while the Delta table in development is a regular Delta table. Time travel doesn’t work well with streaming tables and materialized views.

Smriti2 · a week ago

I check the detail of both the table, it is pointing to the same location and same table id for both the workspace and the format is delta for that table no matter which workspace I query from

pradeep_singh · Tuesday

SO you have the same cloud storage location as the location for this table . So its essentially the same metastore , same catalog , same schema , same location , same credentials . The only thing thats diffrent is workspace ?