I recently ran into an interesting Databricks behavior while implementing a row-count comparison using Delta Time Travel (VERSION AS OF).
Platform: Azure
Scenario:
- Same Unity Catalog
- Same fully qualified table
- Same table ID, location, and Delta format
Yet the behavior differed across environments.
What worked in Dev
- I ran the notebook interactively
- Using an all-purpose cluster
- Delta Time Travel (VERSION AS OF) worked as expected
What failed in Prod
- The same notebook ran as a scheduled Job
- Executed on a job cluster on prod workspace with scheduled job that has one task with a notebook
- The exact same Delta table failed with:
TIME TRAVEL is not allowed. Operation not supported on Streaming Tables
The surprising part
The table itself was unchanged:
- Same catalog
- Same location
- Same Delta properties
- Same table ID
My code compares active row counts between the last two Delta versions of a table, and fails if the row count drops more than 15%, using Delta time travel (VERSION AS OF) to read past snapshots.
