Hi @anurag_dev
Databricks runs a commit service that coordinates writes to Amazon S3 from multiple clusters. This service runs in the Databricks control plane. (441371677306)
The S3 commit service helps guarantee consistency of writes across multiple clusters on a single table in specific cases. For example, the commit service helps Delta Lake implement ACID transactions.
In the default configuration, Databricks sends temporary AWS credentials from the data plane to the control plane in the commit service API call. Instance profile credentials are valid for six hours.
If you use AWS GuardDuty and you access data using AWS IAM instance profiles, GuardDuty may create alerts for default Databricks behavior related to Delta Lake, Structured Streaming, Auto Loader, or COPY INTO. These alerts are related to instance credential exfiltration detection, which is enabled by default. These alerts include the title UnauthorizedAccess: IAMUser/InstanceCredentialExfiltration.InsideAWS.
You can configure your Databricks deployment to address GuardDuty alerts related to the S3 commit service by creating an AWS instance profile that assumes the role of your original S3 data access IAM role.
As an alternative to using instance profile credentials, this new instance profile can configure clusters to assume a role with short-duration tokens. This capability already exists in all recent Databricks Runtime versions and can be enforced globally via cluster policies.
Please refer to the following documentation for a detailed explanation:- https://docs.databricks.com/administration-guide/cloud-configurations/aws/s3-commit-service.html#aws...
