Despite following the steps mentioned in the provided link to create an instance profile, we encountered a problem in step 6 where we couldn't successfully add the instance profile to Databricks(Step 6: Add the instance profile to Databricks).
https://docs.databricks.com/en/connect/storage/tutorial-s3-instance-profile.html
And the error is as follows:
Verification of the instance profile failed. AWS error: You are not authorized to perform this operation. User: arn:aws:sts::755231362028:assumed-role/databricks_role_datacoe/databricks is not authorized to perform: ec2:RunInstances on resource: arn:aws:ec2:us-east-1:755231362028:network-interface/* with an explicit deny in a service control policy. Encoded authorization failure message: IaOFN16u287L-3WIfyxtLvYu1YeV0bOlemGjkeeGYvsA7YDhSXNPq4x7Ei_s_WLKZWZEnJIinOCyUD6WxG9KYx4stIXCmmPXHipYpcUVkVZTC_MEyGlVWWoO57GQ4jtP7Nnle87jCHgWUWWirhYZwYqwTmPfGgzMT0jVhW0Jny4OO_F4juU7OZHlCwXYdVCrC3wvYBVxYJhzqbajIugz_57VmyblPDvjmFt4syGsPbHZDYUL5w7rqz7UMTVlQ5Ge5DjNOts1ZetKb2sQCKAu0akGM6KoHIGq-vYMStvIHCpnRpc_GKbjtUzVQOE7_5UrSYE9EqAp-PY4b5uLzuWtTs7CfY9AABpCFmcoT42Q2ccxESxm_iouzu4B6DQMgAWsd06cSvaIPsIdRiOwV3a-OYrKfgestTlGHUz28F0GMpH_NZf-Gq6F_X4LhLNpmKAETit56MQ9L1Vfjl0EljRWMDircneesYxYC9Y2caN7l247yzAZ57eI-1ck7vbBj11fvGnHPNX3egyfykDs164WPDcHjX3JovEa22vSnAgU_CnS8_rUyGvhbvGQvzepzUSkw4TNx9McDoE0T9tPWxQ7ZAdsxsJgqvLL9cv5buDMVpN27t8Mjajk_YQUQpQ5iBlLsyxrQWZyzT-4jV73eBLDLMPYL5yRNLYsCScrBdQfJUNq5kb2OAVQwlDpD323nrFKyVgXAGRanA5hsHjwS-3_msh4WjhS1pMQsOUSrZc4oqjLmfd-vut2Fgze-xd09S9GeRDnva5-KFpmLhodD6rdMmh2
Show less
In rare cases, the validation error is due to an AWS unavailability or misconfiguration, rather than an actual permission issue. In this case, you can skip the validation and forcibly add an instance profile by selecting "skip validation". If you do this, please try creating a cluster with the profile to make sure it works properly.
Can anyone please help me with this.
NOTE: Databricks Workspace is on AWS
