topic Multi-cluster write for delta tables with s3 as the datastore in Data Engineering

Multi-cluster write for delta tables with s3 as the datastore

hari — Wed, 23 Feb 2022 13:35:27 GMT

Does Delta currently support multi-cluster writes to delta table in s3?

I see in the data bricks documentation that data bricks doesn't support writing to the same table from multiple spark drivers and thus multiple clusters.

But s3Guard was also added to the s3a client for s3 which provides string write consistency for most cases.

Can someone confirm whether delta currently supports multi-cluster writes or are there still edge cases in s3Guard making multi-cluster writes not possible?

Re: Multi-cluster write for delta tables with s3 as the datastore

Hubert-Dudek — Wed, 23 Feb 2022 13:44:43 GMT

Yes it supports. S3 originally doesn't support it (that's why Azure dla is better) but it is handled by AWS Commit. Here is more info https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html

Re: Multi-cluster write for delta tables with s3 as the datastore

hari — Wed, 23 Feb 2022 16:24:43 GMT

Hi @Hubert Dudek ,

Thanks for the confirmation

I didn't know that data bricks had an s3 commit service that ensures atomicity for writes across clusters.

Do you know if commit service is enabled by default

Re: Multi-cluster write for delta tables with s3 as the datastore

nastasiya09 — Wed, 23 Feb 2022 19:53:40 GMT

that's really good post for me

mobdro

verizon wifi