Hi @Yahya24 ,
When you change the cluster access mode to "Shared" on Databricks, the cluster is associated with a Databricks-managed IAM role that is used to access AWS resources. This role might not have the necessary permissions to access DBFS resources in your account.
To access DBFS resources from your code on a cluster with "Shared" access mode, you will need to grant the IAM role associated with the cluster with the necessary permissions to access the DBFS resources.
You can do this by using the AWS Console, or by programmatically creating an IAM policy that grants access to the required DBFS resources and attaching the policy to the role.
Here's an example IAM policy that grants the necessary permissions to access DBFS resources:
%json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToDbfs",
"Effect": "Allow",
"Action": [
"s3:*",
"dynamodb:*"
],
"Resource": [
"arn:aws:s3:::*",
"arn:aws:dynamodb:*:*:table/*"
]
}
]
}
This policy grants access to all S3 and DynamoDB resources in all regions. If you want to limit access to specific resources or regions, you will need to modify the policy accordingly.
After creating the policy, you will need to attach the policy to the IAM role associated with your Databricks cluster. To do this, you can use the AWS CLI or the AWS Console.
Once the IAM role has been granted the necessary permissions, you should be able to access the required DBFS resources from your Python code in the Databricks cluster with "Shared" access mode.
