Attach instance profile to service principal.

[Orianh]

Hey Guys,

I'm having some permission issues using service principal and instance profile and i hope you could help me.

I created a service principal and attached to it an instance profile - databricks-my-profile.

I have a s3 bucket with policy that allow read/write only to service principal databricks-my-profile. this bucket has been mount into dbfs.

I have a cluster with databricks-my-profile instance profile.

While im able to read & write into this s3 bucket from databricks environment( from notebooks, jobs) which is good since the cluster have an instance profile that fits with the s3 bucket restrictions, I can't read & write data from this bucket using my service principal but i can see in its roles that databricks-my-profile exists for this specific sp.

I tried to copy files into the bucket using databricks cli and with the sp token and got an error.

Command use to upload files:

databricks fs ls dbfs:/mnt/my_mounted_bucket --profile my-service-principal

Error i get after runnnig the command:

Error: Authorization failed. Your token may be expired or lack the valid scope

Does some one have any idea why this is failing? or how i should debug this issue?

I check the s3 bucket policy and the restriction are only on instance profile - so this don't happening because ip restrictions or something like this.

Hope you can help me.

Thanks!

[Debayan]

Hi, Could you please check if these were followed:

https://docs.databricks.com/administration-guide/users-groups/service-principals.html

https://docs.databricks.com/aws/iam/instance-profile-tutorial.html

https://docs.databricks.com/aws/iam/manage-instance-profiles.html

Please let us know if this helps.

[Kaniz]

Hi @orian hindi​  (Customer)​, We haven’t heard from you since the last response from @Debayan Mukherjee​, and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

[Orianh · Accepted solution]

Hey @Kaniz Fatma​ , @Debayan Mukherjee​,

Thanks for your answers.

Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.

I'm not sure if this exists in docs (might miss it) but this info can be achieved using debug flag (--debug) on the cli command that i specified...