cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Attach instance profile to service principal.

Orianh
Valued Contributor II

Hey Guys,

I'm having some permission issues using service principal and instance profile and i hope you could help me.

I created a service principal and attached to it an instance profile - databricks-my-profile.

I have a s3 bucket with policy that allow read/write only to service principal databricks-my-profile. this bucket has been mount into dbfs.

I have a cluster with databricks-my-profile instance profile.

While im able to read & write into this s3 bucket from databricks environment( from notebooks, jobs) which is good since the cluster have an instance profile that fits with the s3 bucket restrictions, I can't read & write data from this bucket using my service principal but i can see in its roles that databricks-my-profile exists for this specific sp.

I tried to copy files into the bucket using databricks cli and with the sp token and got an error.

Command use to upload files:

databricks fs ls dbfs:/mnt/my_mounted_bucket --profile my-service-principal

Error i get after runnnig the command:

Error: Authorization failed. Your token may be expired or lack the valid scope

Does some one have any idea why this is failing? or how i should debug this issue?

I check the s3 bucket policy and the restriction are only on instance profile - so this don't happening because ip restrictions or something like this.

Hope you can help me.

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Orianh
Valued Contributor II

Hey @Kaniz Fatma​ , @Debayan Mukherjee​,

Thanks for your answers.

Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.

I'm not sure if this exists in docs (might miss it) but this info can be achieved using debug flag (--debug) on the cli command that i specified...

View solution in original post

2 REPLIES 2

Debayan
Esteemed Contributor III

Orianh
Valued Contributor II

Hey @Kaniz Fatma​ , @Debayan Mukherjee​,

Thanks for your answers.

Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.

I'm not sure if this exists in docs (might miss it) but this info can be achieved using debug flag (--debug) on the cli command that i specified...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group