topic Re: Attach instance profile to service principal. in Data Engineering

Attach instance profile to service principal.

Orianh — Tue, 10 Jan 2023 16:05:09 GMT

Hey Guys,

I'm having some permission issues using service principal and instance profile and i hope you could help me.

I created a service principal and attached to it an instance profile - databricks-my-profile.

I have a s3 bucket with policy that allow read/write only to service principal databricks-my-profile. this bucket has been mount into dbfs.

I have a cluster with databricks-my-profile instance profile.

While im able to read & write into this s3 bucket from databricks environment( from notebooks, jobs) which is good since the cluster have an instance profile that fits with the s3 bucket restrictions, I can't read & write data from this bucket using my service principal but i can see in its roles that databricks-my-profile exists for this specific sp.

I tried to copy files into the bucket using databricks cli and with the sp token and got an error.

Command use to upload files:

databricks fs ls dbfs:/mnt/my_mounted_bucket --profile my-service-principal

Error i get after runnnig the command:

Error: Authorization failed. Your token may be expired or lack the valid scope

Does some one have any idea why this is failing? or how i should debug this issue?

I check the s3 bucket policy and the restriction are only on instance profile - so this don't happening because ip restrictions or something like this.

Hope you can help me.

Thanks!

Re: Attach instance profile to service principal.

Orianh — Tue, 17 Jan 2023 09:40:55 GMT

Hey @Kaniz Fatma , @Debayan Mukherjee,

Thanks for your answers.

Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.

I'm not sure if this exists in docs (might miss it) but this info can be achieved using debug flag (--debug) on the cli command that i specified...

Re: Attach instance profile to service principal.

Debayan — Wed, 11 Jan 2023 22:38:25 GMT

Hi, Could you please check if these were followed:

https://docs.databricks.com/administration-guide/users-groups/service-principals.html

https://docs.databricks.com/aws/iam/instance-profile-tutorial.html

https://docs.databricks.com/aws/iam/manage-instance-profiles.html

Please let us know if this helps.