Same Delta Table, Different Behavior: Dev vs Prod Workspace in Databricks

Smriti2
New Contributor II

I recently ran into an interesting Databricks behavior while implementing a row-count comparison using Delta Time Travel (VERSION AS OF).

Platform: Azure
Scenario:

  • Same Unity Catalog
  • Same fully qualified table
  • Same table ID, location, and Delta format

Yet the behavior differed across environments.

What worked in Dev

  • I ran the notebook interactively
  • Using an all-purpose cluster
  • Delta Time Travel (VERSION AS OF) worked as expected

What failed in Prod

  • The same notebook ran as a scheduled Job
  • Executed on a job cluster on prod workspace with scheduled job that has one task with a notebook
  • The exact same Delta table failed with:

 

TIME TRAVEL is not allowed. Operation not supported on Streaming Tables

The surprising part 

The table itself was unchanged:

  • Same catalog
  • Same location
  • Same Delta properties
  • Same table ID

My code compares active row counts between the last two Delta versions of a table, and fails if the row count drops more than 15%, using Delta time travel (VERSION AS OF) to read past snapshots.