emma_s
Databricks Employee
Databricks Employee

Hi, so I think this is happening as the write to delta log is a different AWS permission, to the put object permission that allows you to the parquet file. My suspicion is some kind of permission update must have happened in the background. The best course of action would be to look at CloudTrail to see exactly what policy is denying your request, the request id is in your logs above. If you don't have cloud trail you can try running the job with this config changed  spark.databricks.delta.commitProtocol.enabled false. This will allow you to see if it is a multithread persmision issue or not.