DatabricksThrottledException Error

[chakradhar545]

Hi,

Our scheduled job runs into below error once in a while and job fails. Any leads or thoughts please why we run into this once in a while and how to fix it please

shaded.databricks.org.apache.hadoop.fs.s3a.DatabricksThrottledException: Instantiate shaded.databricks.org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Rate exceeded (Service: AWSSecurityTokenService; Status Code: 400; Error Code: Throttling; Request ID: b2ec6de9-fd09-4566-894f-750070f11f04; Proxy: null)

[Kaniz]

Hi @chakradhar545, The error message you’re encountering indicates a throttling issue when interacting with Amazon S3 using Databricks.

Let’s break down the error and explore potential solutions:

1. Error Details:

   • The error message mentions two key components:
     • shaded.databricks.org.apache.hadoop.fs.s3a.DatabricksThrottledException: This exception is specific to Databricks and indicates throttling.
     • com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: This exception is from AWS Security Token Service (STS) and indicates that the rate limit has been exceeded.
   • The error code Throttling and the associated Request ID provide additional context.

2. Possible Causes:

   • Rate Limits: AWS imposes rate limits on various API operations, including STS. If your job makes frequent requests to assume roles or obtain temporary credentials, it might hit these limits.
   • Incorrect Configuration: Ensure that your AWS credentials, roles, and permissions are correctly configured in Databricks. Any misconfiguration can lead to authorization issues.
   • Network or Latency Issues: Network latency or transient issues can cause throttling. Check for network stability and retry mechanisms.

3. Steps to Resolve:

   • Review AWS IAM Roles and Permissions:
     • Verify that the IAM role associated with your Databricks cluster has the necessary permissions to assume other roles and interact with S3.
     • Ensure that the role has the sts:AssumeRole permission.
   • Check Rate Limits:
     • Monitor the rate at which your job is making requests to AWS services. If it’s consistently high, consider optimizing your code or batching requests.
     • Implement retries with exponential backoff to handle throttling gracefully.
   • Examine Network and Latency:
     • Investigate any network issues between Databricks and AWS.
     • Consider using a VPC endpoint for S3 to reduce network latency.
   • Verify SAS Key and Permissions:
     • If you’re using a shared access signature (SAS) key for authentication, ensure it has the necessary permissions for the S3 bucket.
     • Double-check that the SAS key is valid and hasn’t expired.
   • Upgrade Databricks Runtime:
     • Sometimes issues are resolved in newer Databricks runtime versions. Consider upgrading to the latest stable version.
   • Review Databricks Configuration:
     • Check if there are any custom configurations related to S3 or AWS in your Databricks environment.
     • Compare your configuration with other working clusters.

4. Debugging:

   • To get more insights, expand the error stack trace and look for lines like “Caused by.” These lines often provide additional context about the root cause.
   • Investigate the specific request ID (b2ec6de9-fd09-4566-894f-750070f11f04) to understand the exact operation that triggered the throttling.

¹: Stack Overflow: shaded.databricks.org.apache.hadoop.fs.azure.AzureException

²: Stack Overflow: Real error - "This request is not authorized to perform this operation using this pe...