topic Delta Lake S3 multi-cluster writes - DynamoDB in Administration & Architecture

Delta Lake S3 multi-cluster writes - DynamoDB

JonLaRose — Tue, 26 Sep 2023 13:18:38 GMT

Hi there!

I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.

I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in the control plane compute or the user's?

If it's in the data plane, how can I provide permissions to create/update this specific table?

If it's in the control plane compute, why is it failing with the following:

Py4JJavaError: An error occurred while calling o476.save. : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: The security token included in the request is invalid.

Thanks!

Re: Delta Lake S3 multi-cluster writes - DynamoDB

JonLaRose — Sun, 08 Oct 2023 10:51:30 GMT

Thank you @Retired_mod .

Does the S3 Commit service use the `s3a` configured S3 endpoint (from the Spark session Hadoop configurations)? If not, is there a way to configure the S3 endpoint that the S3 Commit service uses?

Re: Delta Lake S3 multi-cluster writes - DynamoDB

prem14f — Fri, 26 Jul 2024 18:08:22 GMT

Hi, could you please help me here? How can i use this configuration in DataBricks?
So I will maintain my transcription logs there, and with Parallel, I can use the Delta-RS job.

spark.conf.set("spark.delta.logStore.s3a.impl", "io.delta.storage.S3DynamoDBLogStore")

spark.conf.set("spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName", "delta_log")

spark.conf.set("spark.io.delta.storage.S3DynamoDBLogStore.ddb.region", "eu-west-1")