cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

getting alerts in AWS UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.InsideAWS

anurag_dev
New Contributor

Hi folks, 
I have been using IAM instance profile as a credentials provider to Databricks workspace. I am getting this error 
"Credentials created exclusively for an EC2 instance using instance role <role-arn> have been used from a remote AWS account 414351767826."

Can someone suggest how to tackle it . 
I have tried this link . 

1 REPLY 1

Yeshwanth
Databricks Employee
Databricks Employee

Hi @anurag_dev 

Databricks runs a commit service that coordinates writes to Amazon S3 from multiple clusters. This service runs in the Databricks control plane. (441371677306)

The S3 commit service helps guarantee consistency of writes across multiple clusters on a single table in specific cases. For example, the commit service helps Delta Lake implement ACID transactions.
In the default configuration, Databricks sends temporary AWS credentials from the data plane to the control plane in the commit service API call. Instance profile credentials are valid for six hours.

If you use AWS GuardDuty and you access data using AWS IAM instance profiles, GuardDuty may create alerts for default Databricks behavior related to Delta Lake, Structured Streaming, Auto Loader, or COPY INTO. These alerts are related to instance credential exfiltration detection, which is enabled by default. These alerts include the title UnauthorizedAccess: IAMUser/InstanceCredentialExfiltration.InsideAWS.

You can configure your Databricks deployment to address GuardDuty alerts related to the S3 commit service by creating an AWS instance profile that assumes the role of your original S3 data access IAM role.

As an alternative to using instance profile credentials, this new instance profile can configure clusters to assume a role with short-duration tokens. This capability already exists in all recent Databricks Runtime versions and can be enforced globally via cluster policies.

Please refer to the following documentation for a detailed explanation:- https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html#aws...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group