"AWS S3 resource has been disabled" error on job, not appearing on notebook

[jvk]

I am getting an "INTERNAL_ERROR" on a databricks job submitted through the API. Which says:

"Run result unavailable: run failed with error message All access to AWS S3 resource has been disabled"

However, when I click on the notebook created by the job and run this through the same cluster there are no errors and I am able to access the data.

Why would this be the case?Thanks

[Kaniz_Fatma]

Hi @jvk, The “INTERNAL_ERROR” you’re encountering in your Databricks job, along with the message “Run result unavailable: run failed with error message All access to AWS S3 resource has been disabled,” indicates that there’s an issue related to accessing AWS S3 resources. 

1. AWS Credentials Configuration:

   • Ensure that the AWS credentials (access key and secret key) are correctly configured for your Databricks job. If the credentials are missing or incorrect, it can lead to the “access disabled” error.
   • Verify that the same credentials are being used when you run the notebook manually. Sometimes, different credentials might be set in different contexts, causing discrepancies.

2. IAM Roles vs. AWS Keys:

   • Consider using IAM roles instead of AWS keys for accessing S3 resources. IAM roles provide a more secure and manageable way to grant permissions.
   • If you’re switching from AWS keys to IAM roles, unmount any DBFS mount points associated with S3 buckets created using AWS keys. Then, remount them using the IAM role.
   • Avoid using global init scripts to set AWS keys; instead, use cluster-scoped init scripts if necessa...

3. Instance Profiles and IAM Role Passthrough:

   • When using instance profiles (IAM roles), ensure that the correct IAM role is configured. If you’re using IAM Role Passthrough, verify that the meta instance profile is correctly set up².
   • If you’re using a regular instance profile, make sure that IAM Role Passthrough is disabled.
4. Sometimes, issues with the native Parquet reader can cause S3-related errors. You can work around this problem by disabling the native Parquet reader in your Spark configuration:

   spark.conf.set("spark.databricks.io.parquet.nativeReader.enabled", False)
   

5. This might help resolve any compatibility issues with S3 data access.

         Please review your AWS setup, credentials, and configurations.

[jvk]

Hi @Kaniz_Fatma , thanks for your help. I will look into these issues. 

[jvk]

Hi @Kaniz_Fatma , I have tried you suggestions but no luck.

It works fine when I am running via notebook, but when I put that same notebook into a job run and run that, I get the error:

"Run result unavailable: run failed with error message Cannot access AWS bucket"

[jvk]

@Kaniz_Fatma 

In the s3 logs of the run, I am seeing this:

24/05/29 06:39:30 WARN FileSystem: Failed to initialize fileystem dbfs:///: java.io.FileNotFoundException: Bucket user-workspace-s3-bucket does not exist
24/05/29 06:39:30 ERROR DbfsHadoop3: Failed to close FileSystem
java.lang.NullPointerException