Get exceptionTraceId details

[dvmentalmadess]

I'm getting the following error:

module.consumer_stage_catalog.databricks_external_location.catalog: Creating...
╷
│ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please contact your account admin to update the storage credential. PERMISSION_DENIED: Access denied. Cause: 403 Forbidden error from cloud storage provider. exceptionTraceId=[UUID]
│ 
│   with module.consumer_stage_catalog.databricks_external_location.catalog,
│   on .terraform/modules/consumer_stage_catalog/terraform/databricks_catalog.tf line 49, in resource "databricks_external_location" "catalog":
│   49: resource "databricks_external_location" "catalog" {

Where can I find the logs to look up the details for exceptionTraceId? I only see documentation to setup audit logs and billing logs and the docs don't mention exceptions - I don't see schema or examples that include exceptionTraceId. This is the result of a Databricks API call and there's no running cluster involved so I can't check the cluster logs.

[Kaniz]

Hi @dvmentalmadess, The error message you’re encountering indicates that there was an unhandled error in the API call, and the exceptionTraceId can be used to track down the root cause of the issue.

Let’s break it down:

1. Cause: The AWS IAM role associated with your Databricks environment lacks READ permissions on the specified S3 URL (s3://[bucket name]/catalogs).
2. Solution: You need to update the storage credential for the IAM role. Please contact your account admin to address this issue.

Now, regarding the exceptionTraceId, finding detailed logs related to this specific error can be a bit tricky. Here are some steps you can take:

Databricks REST API Reference: Check the Databricks REST API reference documentation. While it doesn’t explicitly mention exceptionTraceId, it provides details on various API endpoints a.... Unfortunately, there’s no direct endpoint for retrieving exception trace logs.

Here’s a relevant thread that might provide insights.

Audit Logs: Although audit logs are primarily for tracking user activity, they might contain relevant information. Ensure that you’ve set up audit logs properly. If the error is related to authentication or authorization, it could be captured in these logs.

Remember that since this error originates from an API call, there won’t be cluster logs to check. Focus on the IAM role permissions and explore the available logs to narrow down the problem. Good luck troubleshooting! 🛠️

[dvmentalmadess]

@Kaniz 

Thanks for your reply. I had hoped there was a way to see the original exeception to retrieve the S3 request id values so I could open an AWS support ticket, if the IAM identity and denied permission weren't already listed in the original exception. After reading this thread which mentioned looking up exceptionTraceId in Databricks logs I had hoped that's where I would find the information I needed.

I was asking after I had already investigated both the IAM resource and identity policies, compared them to existing policies that were functioning as well as to the DBR documentation, and also used the AWS IAM Policy Simulator.

As it so happens, I'm pretty sure I did find the problem after posting this. I'm just waiting for a response to confirm. 

That said, I'd be interested in the relevant thread you mentioned but the link provided just points to the same resource url as the previous link you provided. If you'd be willing to update the post or share the link in a reply I'd love to read more.

If the API team ends up reading this, I'd like to provide the following feedback. Providing the means to access the AWS request and extended request id values would be useful for resolving issues. Especially one like this where the likely cause is a context key that a policy condition relies on. Having the ids required to open up an AWS support case would have allowed me to work with AWS support who are likely to have the context values sent in the request which would have reduced the time to resolution significantly. The only reason I even have an idea why this isn't working is because I happened to notice the External ID value displayed in the list of credentials and that it was different than every other instance.