I am getting error on delta write - com.databricks.s3commit.S3CommitFailedException: Access Denied

[sk1996]

I am facing delta write issue on one of my non prod bucket. I am using correct role and bucket has all the required access. 

When data is being written as parquet, it works fine. But when we change format to delta, it fails with Access Denied error and that too only for delta commit. All parquet files under the hood is being written correctly but it fails to commit the delta log. This started failing suddenly when there has been no changes with code or access. Below is the exact error

com.databricks.s3commit.S3CommitFailedException: Access Denied; request: POST "s3 path" /person/_delta_log/00000000000000008340.json 

aws-sdk-java/1.12.783 Linux/5.4.0-1154-aws-fips Java_HotSpot(TM)_64-Bit_Server_VM/17.0.18+8-LTS-jvmci-23.0-b80 java/17.0.18 scala/2.12.20-bin-db-6-dda2050 kotlin/2.1.0 vendor/Oracle_Corporation cfg/retry-mode/legacy cfg/auth-source#unknown m/P com.amazonaws.services.s3.model.CompleteMultipartUploadRequest; Request ID: B7K1QVD6RD4Y4V63, Extended Request ID: bRaQUf2gf9ZsDzmebmpf4BAuaWjSHL/w2ldpJWRZMW7xrhGmn7b4/P131EiWHw/jTnoHFMZ0ug8=, Cloud Provider: unknown, Instance ID: unknown (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: B7K1QVD6RD4Y4V63; S3 Extended Request ID: bRaQUf2gf9ZsDzmebmpf4BAuaWjSHL/w2ldpJWRZMW7xrhGmn7b4/P131EiWHw/jTnoHFMZ0ug8=; Proxy: null)

[emma_s]

Hi, so I think this is happening as the write to delta log is a different AWS permission, to the put object permission that allows you to the parquet file. My suspicion is some kind of permission update must have happened in the background. The best course of action would be to look at CloudTrail to see exactly what policy is denying your request, the request id is in your logs above. If you don't have cloud trail you can try running the job with this config changed  spark.databricks.delta.commitProtocol.enabled false. This will allow you to see if it is a multithread persmision issue or not.