The short answer is no: independent operations from different jobs become separate, serialized commits in the Delta transaction log. They won’t be coalesced into one commit unless you explicitly run a single statement that performs both (for example, a MERGE/OVERWRITE that rewrites files and inserts rows).

Some practical guidelines:

 

  • Keep ingestion appends and retention deletes as separate statements/jobs so they become separate commits and skipChangeCommits only skips the delete commit

 

  • Avoid MERGE/OVERWRITE that mixes rewrites and inserts in the source Bronze table. If you must, expect the commit to be skipped entirely by skipChangeCommits.
  • If concurrent operations overlap in time, they are still serialized as distinct commits. Streaming reads will see them as separate versions in order.

This blog post does a great job of explaining the delta transaction log: https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html

 

 

View solution in original post