cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DatabricksThrottledException Error

chakradhar545
New Contributor

Hi,

Our scheduled job runs into below error once in a while and job fails. Any leads or thoughts please why we run into this once in a while and how to fix it please

shaded.databricks.org.apache.hadoop.fs.s3a.DatabricksThrottledException: Instantiate shaded.databricks.org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Rate exceeded (Service: AWSSecurityTokenService; Status Code: 400; Error Code: Throttling; Request ID: b2ec6de9-fd09-4566-894f-750070f11f04; Proxy: null)

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @chakradhar545The error message youโ€™re encountering indicates a throttling issue when interacting with Amazon S3 using Databricks.

Letโ€™s break down the error and explore potential solutions:

  1. Error Details:

    • The error message mentions two key components:
      • shaded.databricks.org.apache.hadoop.fs.s3a.DatabricksThrottledException: This exception is specific to Databricks and indicates throttling.
      • com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: This exception is from AWS Security Token Service (STS) and indicates that the rate limit has been exceeded.
    • The error code Throttling and the associated Request ID provide additional context.
  2. Possible Causes:

    • Rate Limits: AWS imposes rate limits on various API operations, including STS. If your job makes frequent requests to assume roles or obtain temporary credentials, it might hit these limits.
    • Incorrect Configuration: Ensure that your AWS credentials, roles, and permissions are correctly configured in Databricks. Any misconfiguration can lead to authorization issues.
    • Network or Latency Issues: Network latency or transient issues can cause throttling. Check for network stability and retry mechanisms.
  3. Steps to Resolve:

    • Review AWS IAM Roles and Permissions:
      • Verify that the IAM role associated with your Databricks cluster has the necessary permissions to assume other roles and interact with S3.
      • Ensure that the role has the sts:AssumeRole permission.
    • Check Rate Limits:
      • Monitor the rate at which your job is making requests to AWS services. If itโ€™s consistently high, consider optimizing your code or batching requests.
      • Implement retries with exponential backoff to handle throttling gracefully.
    • Examine Network and Latency:
      • Investigate any network issues between Databricks and AWS.
      • Consider using a VPC endpoint for S3 to reduce network latency.
    • Verify SAS Key and Permissions:
      • If youโ€™re using a shared access signature (SAS) key for authentication, ensure it has the necessary permissions for the S3 bucket.
      • Double-check that the SAS key is valid and hasnโ€™t expired.
    • Upgrade Databricks Runtime:
      • Sometimes issues are resolved in newer Databricks runtime versions. Consider upgrading to the latest stable version.
    • Review Databricks Configuration:
      • Check if there are any custom configurations related to S3 or AWS in your Databricks environment.
      • Compare your configuration with other working clusters.
  4. Debugging:

    • To get more insights, expand the error stack trace and look for lines like โ€œCaused by.โ€ These lines often provide additional context about the root cause.
    • Investigate the specific request ID (b2ec6de9-fd09-4566-894f-750070f11f04) to understand the exact operation that triggered the throttling.

1: Stack Overflow: shaded.databricks.org.apache.hadoop.fs.azure.AzureException

2: Stack Overflow: Real error - "This request is not authorized to perform this operation using this pe...

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group