Delta Lake S3 multi-cluster writes - DynamoDB

[JonLaRose]

Hi there!

I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.

I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in the control plane compute or the user's?

If it's in the data plane, how can I provide permissions to create/update this specific table?

If it's in the control plane compute, why is it failing with the following: 

Py4JJavaError: An error occurred while calling o476.save.
: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: The security token included in the request is invalid.

?

Thanks!

[JonLaRose]

Thank you @Retired_mod .

Does the S3 Commit service use the `s3a` configured S3 endpoint (from the Spark session Hadoop configurations)? If not, is there a way to configure the S3 endpoint that the S3 Commit service uses? 

[prem14f]

Hi, could you please help me here? How can i use this configuration in DataBricks? 
So I will maintain my transcription logs there, and with Parallel, I can use the Delta-RS job.

spark.conf.set("spark.delta.logStore.s3a.impl", "io.delta.storage.S3DynamoDBLogStore")

spark.conf.set("spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName", "delta_log")

spark.conf.set("spark.io.delta.storage.S3DynamoDBLogStore.ddb.region", "eu-west-1")