Multi-cluster write for delta tables with s3 as the datastore

[hari]

Does Delta currently support multi-cluster writes to delta table in s3?

I see in the data bricks documentation that data bricks doesn't support writing to the same table from multiple spark drivers and thus multiple clusters.

But s3Guard was also added to the s3a client for s3 which provides string write consistency for most cases.

Can someone confirm whether delta currently supports multi-cluster writes or are there still edge cases in s3Guard making multi-cluster writes not possible?

[Hubert-Dudek · Accepted solution]

Yes it supports. S3 originally doesn't support it (that's why Azure dla is better) but it is handled by AWS Commit. Here is more info ​https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html

[hari]

Hi @Hubert Dudek​ ,

Thanks for the confirmation

I didn't know that data bricks had an s3 commit service that ensures atomicity for writes across clusters.

Do you know if commit service is enabled by default

[nastasiya09]

that's really good post for me

mobdro

verizon wifi