article How to use Databricks Autoloader across AWS accounts in Technical Blog

How to use Databricks Autoloader across AWS accounts

MohanaBasak — Tue, 28 May 2024 13:30:20 GMT

Re: How to use Databricks Autoloader across AWS accounts

iddoi — Fri, 26 Jul 2024 08:56:39 GMT

Hello!

Thanks a lot for this excellent and detailed write up. I have tried to configure this myself but for some reason couldn't make it work.

My cluster was able to assume the instance profile in Account B successfully. I was then able to use boto3 in a Notebook to assume the role in Account A and to get the appropriate S3 object from Account A.

sts_client = boto3.client('sts') assumed_role_object = sts_client.assume_role( RoleArn=role_arn, RoleSessionName=session_name, ExternalId=external_id ) # Get the temporary credentials credentials = assumed_role_object['Credentials'] # Create an S3 client using the temporary credentials s3_client = boto3.client( 's3', aws_access_key_id=credentials['AccessKeyId'], aws_secret_access_key=credentials['SecretAccessKey'], aws_session_token=credentials['SessionToken'] ) # Try to get the object response = s3_client.get_object(Bucket=bucket_name, Key=file_key) # Read some of the content content = response['Body'].read(1024) # Read first 1KB print(f"Successfully accessed the file. First 1KB of content:\n{content}")

I then attempted to use spark for this and got an AccessDeniedException:

from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("S3DataReadTest").getOrCreate() # Configure Spark to use AWS IAM role spark.conf.set("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider") spark.conf.set("spark.hadoop.fs.s3a.assumed.role.arn", role_arn) spark.conf.set("spark.hadoop.fs.s3a.assumed.role.session.name", role_session_name) spark.conf.set("spark.hadoop.fs.s3a.assumed.role.external.id", role_external_id) spark.conf.set("spark.hadoop.fs.s3a.assumed.role.sts.endpoint", "sts.amazonaws.com") # Read the data data_df = spark.read.format("csv").load(s3_bucket_path)

I also tried to use the temporary credentials I got from the successful assume_role command in boto3, but that didn't work either:

spark = SparkSession.builder \ .config("spark.hadoop.fs.s3a.access.key", credentials['AccessKeyId']) \ .config("spark.hadoop.fs.s3a.secret.key", credentials['SecretAccessKey']) \ .config("spark.hadoop.fs.s3a.session.token", credentials['SessionToken']) \ .config("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider") \ .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \ .config("spark.hadoop.com.amazonaws.services.s3.enableV4", "true") \ .config("spark.hadoop.fs.s3a.endpoint", "s3.us-east-1.amazonaws.com") \ .getOrCreate() # Attempt to read the file print(f"Attempting to read file: {s3_path}") df = spark.read.csv(s3_path, header=True, inferSchema=True)

Any idea what the issue could be? Any insight would be appreciated! 🙏

* By the way, for some reason the boto3 access worked for me when using a private cluster, but didn't work on a shared cluster (even though the instance profile was assumed in both cases).

When running this on a shared cluster:

# Assume the role sts_client = boto3.client('sts') assumed_role_object = sts_client.assume_role( RoleArn=role_arn, RoleSessionName=session_name, ExternalId=external_id )

I get the following error:

NoCredentialsError: Unable to locate credentials

Re: How to use Databricks Autoloader across AWS accounts

MohanaBasak — Fri, 26 Jul 2024 18:44:46 GMT

@iddoi This blog is specifically for Databricks Autoloader setup using an Instance Profile in Databricks. If you are trying to read data from an S3 bucket (either in the same AWS account, or a different one) to a Databricks notebook, the recommended approach is to use Unity Catalog (UC). With UC, you will not need any instance profile, or need to set any Spark configs. Once you successfully set a connection to an S3 bucket with UC (storage credential and external location), you will directly be able to read from that bucket.

If you are not using UC, and you are trying to connect using an Instance Profile, the bucket in the other AWS account will be accessible if the instance profile permissions are properly set. With this also you will not need any Spark configs if the Instance Profile setup is correct.

That said, I was able to read from an S3 bucket by using your boto3 STS code to get temporary credentials, and then setting hadoop credentials in this way:

aws_access_key_id=credentials['AccessKeyId'] aws_secret_access_key=credentials['SecretAccessKey'] aws_session_token=credentials['SessionToken'] sc._jsc.hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider") sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", aws_access_key_id) sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", aws_secret_access_key) sc._jsc.hadoopConfiguration().set("fs.s3a.session.token", aws_session_token) data_df = spark.read.format("csv").load('s3://acc-a-autol-input/rnd-2022-notes.csv') display(data_df)

I hope this helps. Again, Unity Catalog is the recommended way of reading data from S3 to Databricks.

Re: How to use Databricks Autoloader across AWS accounts

MohanaBasak — Thu, 01 May 2025 15:40:53 GMT

If you have migrated to UC and are looking for a way to set up Auto Loader file Notification in UC-enabled clusters, then read the latest blog here - https://community.databricks.com/t5/technical-blog/how-to-set-up-autoloader-in-file-notification-mode-on-aws-using/ba-p/117048

Re: How to use Databricks Autoloader across AWS accounts

427430 — Thu, 19 Feb 2026 12:25:09 GMT

Hi @MohanaBasak , thank you for the blog post, that's very useful. We're using Databricks autoloader (with notification mode without using UC) to load files across aws accounts. The databricks job is in account B and the s3 bucket / sqs-to-be-created-by-autoloader / sns-to-be-created-by-autoloader are in account A. The only difference is that our databricks job in account B needs to write to account B's S3 and Glue (not writting to account A's S3 as you described in the blog post). I believe that it's a common use case. Do you think after the account B role `arn:aws:iam::<account_b_id>:role/acc_b_instance_profile` has assumed the the account A role `arn:aws:iam::<account_a_id>:role/acc_a_autol_auto_create_role`, the databricks job would lose its original permissions (given to `arn:aws:iam::<account_b_id>:role/acc_b_instance_profile`) to write to account B's S3 and Glue? Or the databricks job is smart enough to only assume the role when using autoloader with SQS/SNS, and skip assuming the role (so using the original `arn:aws:iam::<account_a_id>:role/acc_a_autol_auto_create_role`) when writing to account B's S3 and Glue?

Thank you for your help.

Re: How to use Databricks Autoloader across AWS accounts

MohanaBasak — Mon, 23 Feb 2026 22:05:11 GMT

@427430 I haven't tested this out, but I would say that if the instance profile attached to the cluster has access to write out into the S3 bucket of Acc B, it should be able to. Having said that, please use the UC version of this - https://community.databricks.com/t5/technical-blog/how-to-set-up-autoloader-in-file-notification-mode-on-aws-using/ba-p/117048. UC is the Databricks recommended approach to interact with any cloud storage, and yes, if you use this approach, you should be able to write to a bucket in Acc B.