Unity Catalog external tables

[JonLaRose]

What are the consistency guarantees that DataBricks supply for multi writers, given that the written table is an external table?

Are they different from the consistency guarantees given for managed tables?

Thanks!

[Ajay-Pandey]

Hi @JonLaRose 

Databricks provides specific consistency guarantees depending on whether you're dealing with managed or external tables, particularly in the context of multi-writer scenarios

Key Differences:

• Control Over Storage: Databricks has full control over managed tables, allowing it to enforce stronger consistency guarantees, especially in multi-writer scenarios. In contrast, external tables rely on the external storage system, which may not support the same level of consistency.
• Multi-Writer Scenarios: Managed tables offer better consistency in multi-writer scenarios due to Delta Lake's ACID properties. External tables, on the other hand, are more prone to issues like write conflicts and eventual consistency problems.

[JonLaRose]

Thank you @Ajay-Pandey, that is helpful.

One thing that I'm not sure about is how does DataBricks can use the same ACID mechanism that external tools use with the external tables? For example, if an external Spark cluster write Delta Logs with a LogStore implementation of S3DynamoDBLogStore, and specify a table for all writers, how can DataBricks use this table as well to maintain ACID guarantees?