- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2022 05:35 AM
Does Delta currently support multi-cluster writes to delta table in s3?
I see in the data bricks documentation that data bricks doesn't support writing to the same table from multiple spark drivers and thus multiple clusters.
But s3Guard was also added to the s3a client for s3 which provides string write consistency for most cases.
Can someone confirm whether delta currently supports multi-cluster writes or are there still edge cases in s3Guard making multi-cluster writes not possible?
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2022 05:44 AM
Yes it supports. S3 originally doesn't support it (that's why Azure dla is better) but it is handled by AWS Commit. Here is more info https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2022 05:44 AM
Yes it supports. S3 originally doesn't support it (that's why Azure dla is better) but it is handled by AWS Commit. Here is more info https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2022 08:24 AM
Hi @Hubert Dudek ,
Thanks for the confirmation
I didn't know that data bricks had an s3 commit service that ensures atomicity for writes across clusters.
Do you know if commit service is enabled by default
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2022 11:53 AM

