cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Attach instance profile to service principal.

Orianh
Valued Contributor II

Hey Guys,

I'm having some permission issues using service principal and instance profile and i hope you could help me.

I created a service principal and attached to it an instance profile - databricks-my-profile.

I have a s3 bucket with policy that allow read/write only to service principal databricks-my-profile. this bucket has been mount into dbfs.

I have a cluster with databricks-my-profile instance profile.

While im able to read & write into this s3 bucket from databricks environment( from notebooks, jobs) which is good since the cluster have an instance profile that fits with the s3 bucket restrictions, I can't read & write data from this bucket using my service principal but i can see in its roles that databricks-my-profile exists for this specific sp.

I tried to copy files into the bucket using databricks cli and with the sp token and got an error.

Command use to upload files:

databricks fs ls dbfs:/mnt/my_mounted_bucket --profile my-service-principal

Error i get after runnnig the command:

Error: Authorization failed. Your token may be expired or lack the valid scope

Does some one have any idea why this is failing? or how i should debug this issue?

I check the s3 bucket policy and the restriction are only on instance profile - so this don't happening because ip restrictions or something like this.

Hope you can help me.

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Orianh
Valued Contributor II

Hey @Kaniz Fatma​ , @Debayan Mukherjee​,

Thanks for your answers.

Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.

I'm not sure if this exists in docs (might miss it) but this info can be achieved using debug flag (--debug) on the cli command that i specified...

View solution in original post

3 REPLIES 3

Debayan
Esteemed Contributor III
Esteemed Contributor III

Kaniz
Community Manager
Community Manager

Hi @orian hindi​  (Customer)​, We haven’t heard from you since the last response from @Debayan Mukherjee​, and I was checking back to see if his suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Orianh
Valued Contributor II

Hey @Kaniz Fatma​ , @Debayan Mukherjee​,

Thanks for your answers.

Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.

I'm not sure if this exists in docs (might miss it) but this info can be achieved using debug flag (--debug) on the cli command that i specified...

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!