cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

faisal_da
by New Contributor II
  • 1235 Views
  • 7 replies
  • 2 kudos

Unable to see Manage Account option in the Databricks Workspace

Hi, I have an organizational account who is the owner of the databricks workspace (premium) and also the global administrator. Still, I don't see "Account Console" option in the databricks after clicking the "manage account" option.I have tried to cl...

  • 1235 Views
  • 7 replies
  • 2 kudos
Latest Reply
naikashok
Visitor
  • 2 kudos

Hi @szymon_dybczak ,Yes have databricks premium account and admin as well

  • 2 kudos
6 More Replies
deepud26
by New Contributor
  • 85 Views
  • 2 replies
  • 1 kudos

Connectivity of dbx serverless to api gateway dns in aws is failing

Hello, I'm trying to establish the connection between dbx serverless account which is in "Account A" to the api gateway dns name which is in "Account B" is failing. i have created a api gateway with the custom domain for ex : "api-dev.amazon.com" and...

  • 85 Views
  • 2 replies
  • 1 kudos
Latest Reply
deepud26
New Contributor
  • 1 kudos

Hello @ilir_nuredini ,I'm trying the below way import requestsimport json API_URL = 'https://api-dev.amazon.com/v1/quality'headers = {    "Authorization": f"Bearer {token}",    "Content-Type": "application/json"}response = requests.get(API_URL, heade...

  • 1 kudos
1 More Replies
Lovebo
by New Contributor
  • 212 Views
  • 1 replies
  • 2 kudos

Error creating Databricks workspace on AWS (CREATE_FAILED with JSONDecodeError)

Hi everyone,I’m trying to create a Databricks workspace on AWS by following the official quick start guide:https://docs.databricks.com/aws/en/admin/workspace/quick-startHowever, my CloudFormation stack fails with the following error:  The resource cr...

  • 212 Views
  • 1 replies
  • 2 kudos
Latest Reply
BS_THE_ANALYST
Honored Contributor III
  • 2 kudos

Hi @Lovebo , I'm not too familiar with AWS but happy to have a go at helping. Let's see if we can rule out some of the more obvious things which are easy to miss on the documentation.First of all, are you just following the guide through: https://doc...

  • 2 kudos
noorbasha534
by Valued Contributor
  • 70 Views
  • 2 replies
  • 0 kudos

Configure job access_control_list block from a single place outside of job definition

Hi allIs it possible to configure the access_control_list block ((basically that contains permissions)) from a single place outside of Databricks job definition? This is changing as we are re-defining our permissions model, and each time resulting in...

  • 70 Views
  • 2 replies
  • 0 kudos
Latest Reply
Pilsner
Contributor
  • 0 kudos

Hello @noorbasha534 If you are trying to alter the permissions across many jobs at scale, I believe there are a couple of options to help speed up the process. Firstly, creating groups of users should help, as you can then easily change an entire gro...

  • 0 kudos
1 More Replies
sunykim
by New Contributor
  • 206 Views
  • 3 replies
  • 0 kudos

databricks terraform provider, databricks_credential resource, service

I cannot make the databricks_credential resource create a service credential. It works fine with storage credentials. However, when i put `purpose = "SERVICE"` plus aws_iam_role and comment, in the apply phase it fails with `Error: cannot create cred...

  • 206 Views
  • 3 replies
  • 0 kudos
Latest Reply
sunykim
New Contributor
  • 0 kudos

I have the same error message now when trying to create a USE_SCHEMA grant for a service principal as in https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grant#schema-grants . I create a new service principal and th...

  • 0 kudos
2 More Replies
erigaud
by Honored Contributor
  • 26771 Views
  • 13 replies
  • 11 kudos

Resolved! Installing libraries on job clusters

Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ? 

  • 26771 Views
  • 13 replies
  • 11 kudos
Latest Reply
Charlenendoza
New Contributor
  • 11 kudos

Frankly, I'm also scratching my head about this library installation conundrum in job clusters; the missing Libraries tab is definitely throwing me off. Seems like init scripts are the only officially sanctioned route, which feels a bit clunky. Are t...

  • 11 kudos
12 More Replies
tariq
by New Contributor III
  • 190 Views
  • 3 replies
  • 0 kudos

Databricks Execution Context scheduling

Databricks allows a cluster to have a maximum of 150 execution context. If a number of execution context is attached and running operations on a databricks cluster with a driver and n workers, then how is the execution scheduled? I am assuming that o...

  • 190 Views
  • 3 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Ok i get it.An execution context is not bound to cpu.  It is like a session.  So basically the limit of 150 execution contexts mean that 150 sessions/spark programs can run simultaneously on the cluster (whether that is possible on the hardware is an...

  • 0 kudos
2 More Replies
samson01
by New Contributor II
  • 178 Views
  • 2 replies
  • 1 kudos

Resolved! AWS Account Level provider "databricks" Authentication

I am trying to deploy an additional workspace to my AWS Account with terraform, I keep running into this authentication an error message below, I have searched everywhere in my account console  and used chatgpt  which keep point me to the account con...

  • 178 Views
  • 2 replies
  • 1 kudos
Latest Reply
samson01
New Contributor II
  • 1 kudos

thanks that works

  • 1 kudos
1 More Replies
Ludo
by New Contributor III
  • 3177 Views
  • 5 replies
  • 5 kudos

Issue with Databricks Alerts formatting sent to Microsoft Teams

Hello all,As of Thursday 20th 2025, we started to have our alert notifications being sent via Microsoft Teams with no formatting (raw HTML).We use a custom template with QUERY_RESULT_TABLE  and a few html tags.Looks like databricks alerts deprecated ...

  • 3177 Views
  • 5 replies
  • 5 kudos
Latest Reply
Ludo
New Contributor III
  • 5 kudos

Hello,Thanks for sharing the explanation and link to the Microsoft's documentation.We had removed the HTML formatting a few months ago.

  • 5 kudos
4 More Replies
Michael_Appiah
by Contributor
  • 2918 Views
  • 4 replies
  • 2 kudos

Enforcing Tags on SQL Warehouses

Is there a way to enforce tags on SQL Warehouses? Regular cluster policies do not apply to SQL Warehouses and budget policies do not cover SQL Warehouses either, which I find quite surprising given the fact that budget policies are documented as "Att...

  • 2918 Views
  • 4 replies
  • 2 kudos
Latest Reply
BS_THE_ANALYST
Honored Contributor III
  • 2 kudos

@Michael_Appiah this isn't any area I'm familiar with but I thought I'd try and get the ball rolling. Have you looked into this https://docs.databricks.com/aws/en/admin/clusters/policy-definition . It could be a red herring from me, but your article ...

  • 2 kudos
3 More Replies
eshwari
by New Contributor II
  • 470 Views
  • 5 replies
  • 5 kudos

Resolved! Unable to Create Cluster in ADW Deployment — CONTROL_PLANE_REQUEST_FAILURE

I'm running into an issue with my Databricks workspace in Azure in my own VNet. I've successfully created two private endpoints: databricks_ui_api and browser_authenticationHowever, when I try to create a cluster, I get the following error:CONTROL_PL...

  • 470 Views
  • 5 replies
  • 5 kudos
Latest Reply
Khaja_Zaffer
Contributor
  • 5 kudos

Hello @eshwari Good day, I think  szymon_dybczak and  me provided enough information, please let me know if you had the solutions.If you found the solution useful, you can select the solution for the solution which helps others.  Have a greate day!

  • 5 kudos
4 More Replies
Swastick
by New Contributor II
  • 142 Views
  • 3 replies
  • 2 kudos

Resolved! GA for AIM

Do anyone know when Automatic Identity Management will be generally available for all the users?Announcing Automatic Identity Management for Azure Databricks | Databricks Blog

  • 142 Views
  • 3 replies
  • 2 kudos
Latest Reply
Swastick
New Contributor II
  • 2 kudos

Thanks for the clarification!I manage a fairly large Databricks environment, so enabling features that are still in Public Preview can be a bit challenging from a governance and stability standpoint. If possible, could you share any tentative timelin...

  • 2 kudos
2 More Replies
noorbasha534
by Valued Contributor
  • 154 Views
  • 1 replies
  • 0 kudos

PATs sharing in a global data platform

Hello allChecking on how others implement sharing of Databricks personal access tokens for authentication wherein you have atleast 25+ different technologies extracting data via SQL warehouses ((imagine a global data platform that hosts data for usag...

Administration & Architecture
authentication
OAuth
PAT
  • 154 Views
  • 1 replies
  • 0 kudos
Latest Reply
Isi
Honored Contributor II
  • 0 kudos

Hey @noorbasha534 Honestly, I really understand your pain around token management. I face the same situation myself and it can definitely become a headache, especially when you have multiple technologies in play, some of them open-source, and even ca...

  • 0 kudos
marcusfox
by New Contributor
  • 542 Views
  • 2 replies
  • 0 kudos

Webhooks for Microsoft Teams are not being received from Databricks jobs

We have configured a couple of webhooks in Teams channels and added the URLs to Databricks under > Settings> notifications. But our jobs do not post anything into the Teams channels.This used to work but is now not doing anything.

  • 542 Views
  • 2 replies
  • 0 kudos
Latest Reply
daidaidi
New Contributor II
  • 0 kudos

Hello marcusfox, I encounter the same issue as u did. If u already solved it, could u please give the tutorial here please? Thank u in advance for guideline

  • 0 kudos
1 More Replies