balajij8
Esteemed Contributor

@databrciks 

This situation is now elegantly solved using Multi Table Transactions in Databricks. Wrap the Truncate & Load logic in an ATOMIC block and you can achieve beyond single table consistency. This ensures that even if you are performing a truncate and load, report readers see the version before it.

Reports will not encounter an empty table or a partial state. Reports will see the valid version after truncate & full load is done. If the load fails, the transaction rolls back automatically leaving the reports stable.

More details here

View solution in original post