cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

lasmali
by New Contributor II
  • 1892 Views
  • 3 replies
  • 0 kudos

Instance Profile creation via the Databricks REST API returns "INVALID_PARAMETER_VALUE"

# Problem StatementWe have a need to create Instance Profiles via the Databricks REST API, but the endpoint returns an "INVALID_PARAMETER_VALUE" error with "Syntactically invalid AWS instance profile ARN" message even when provided an appropriate ARN...

  • 1892 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @lasse Lidegaard​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
2 More Replies
Orianh
by Valued Contributor II
  • 3518 Views
  • 2 replies
  • 1 kudos

Resolved! Attach instance profile to service principal.

Hey Guys, I'm having some permission issues using service principal and instance profile and i hope you could help me.I created a service principal and attached to it an instance profile - databricks-my-profile.I have a s3 bucket with policy that all...

  • 3518 Views
  • 2 replies
  • 1 kudos
Latest Reply
Orianh
Valued Contributor II
  • 1 kudos

Hey @Kaniz Fatma​ , @Debayan Mukherjee​, Thanks for your answers.Actually, Databricks is not support using DBFS API with service principal & attached instance profile on a mounted s3 bucket.I'm not sure if this exists in docs (might miss it) but thi...

  • 1 kudos
1 More Replies
JasonN
by New Contributor II
  • 1833 Views
  • 2 replies
  • 2 kudos

Resolved! DLT Cluster accessing to S3 bucket without Instance Profile attached

Hi Team,Can anyone please help me figure out how to configure Delta Live Tables Cluster accessing AWS S3 bucket without Instance profile defined in Cluster's JSON?The idea is, the user who is running the DLT Cluster has Storage Credentials and Extern...

  • 1833 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 2 kudos

Hi @Jason Nam​ , DLT and unity catalog are not integrated yet. The cluster-notebook setup uses UC and can access S3 but not the DLT jobs. Please check the limitations in this document (7th point):https://docs.databricks.com/release-notes/unity-catalo...

  • 2 kudos
1 More Replies
mortenhaga
by Contributor
  • 3142 Views
  • 4 replies
  • 4 kudos

Resolved! SQL Serverless Endpoint failing to start with Instance Profile

Hi allSuper stoked about the PP of SQL Serverless, but it does seem that the instance profile Im using doesnt have the required trust relationship for it to work with the Sererless Endpoint. Altough on "classic" mode it works fine. Does Serverless re...

image
  • 3142 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Thank you for sharing your valuable solution, it's work properly.mymilestonecard

  • 4 kudos
3 More Replies
Juniper_AIML
by New Contributor
  • 2500 Views
  • 1 replies
  • 0 kudos

How to setup Instance profile for initializing Databricks Cluster using Docker?

I was trying to start of the Databricks cluster through a docker image. I followed the setup instruction. Excluding the additional setup to setup the IAM role and instance profile as I was facing issues.The image is stored on AWS ECR in a public repo...

Screenshot 2022-02-15 at 2.39.57 PM Screenshot 2022-02-15 at 2.49.03 PM
  • 2500 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hello again, @Aman Gaurav​. Thanks for your second question. As before, we'll see what the community has to say first.

  • 0 kudos
User16752239289
by Valued Contributor
  • 2281 Views
  • 1 replies
  • 0 kudos

The cluster with the instance profile cannot access the S3 bucket. 403 permission denied is thrown

The document has been followed to configure the instance profile. The ec2 instance is able to access the S3 bucket when configured the same instance profile. However, the cluster configured to use the same instance profile failed to access the S3 buc...

  • 2281 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16752239289
Valued Contributor
  • 0 kudos

I suspect this is due to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY has been added to the spark environmental variable. You can run %sh env | grep -i aws on your cluster and make sure AWS_ACCESS_KEY_ID is not present. If so, then please remove it e...

  • 0 kudos
Labels