- 2080 Views
- 3 replies
- 0 kudos
Move metastore to another azure subscription
Hi, We need to migrate our metastore with Unity Catalog to a new Azure subscription while remaining in the same Azure region. Currently, we have two workspaces attached to a single Unity Catalog. I’m looking for the best approach to move the metastor...
- 2080 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @alesventus,There are few points to be considered before migrating from One metastore to another. We need to see how the catalogs, schemas and tables are created as of now.If you have created everything has managed like managed catalog, schema and...
- 0 kudos
- 507 Views
- 4 replies
- 1 kudos
Resolved! REST API for swapping cluster
Hi Team, I am trying to find REST API reference for swapping a cluster but unable to find it in the documentation. Can anyone please tell me what is the REST API reference for swapping an existing cluster to another existing cluster, if present?If no...
- 507 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @szymon_dybczak,It helps! I am able to change the clusters. Thanks a lot!
- 1 kudos
- 2284 Views
- 9 replies
- 6 kudos
Issue accessing databricks secrets from ADF
Hello -Seeing an issue where notebook triggered from ADF is not able to access secret scopes, which was working earlier. Here are the steps I did 1. Provide ADF contributor role permission in databrick workspace. - we tested this and were able to tri...
- 2284 Views
- 9 replies
- 6 kudos
- 6 kudos
Hey @gs2 @IkuyoshiKuroda ,I have reviewed the documentation:According to the Databricks documentation on secret scopes, there are two types:Databricks-backed scopes → secrets are stored inside Databricks. These do not support authentication via Azure...
- 6 kudos
- 755 Views
- 2 replies
- 0 kudos
Resolved! Using pip cache for pypi compute libraries
I am able to configure pip's behavior w.r.t index url by setting PIP_INDEX_URL, PIP_TRUSTED_HOST etc. I would like to cache compute-wide pypi libraries, to improve cluster startup performance / reliability. However, I notice that PIP_CACHE_DIR has no...
- 755 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Isi,We moved away from docker images for the reasons you mention, and because they otherwise had issues for us. We are already using artifactory (as hinted by the environment variables mentioned in my post). I wanted to try further improving the s...
- 0 kudos
- 1976 Views
- 7 replies
- 2 kudos
Unable to see Manage Account option in the Databricks Workspace
Hi, I have an organizational account who is the owner of the databricks workspace (premium) and also the global administrator. Still, I don't see "Account Console" option in the databricks after clicking the "manage account" option.I have tried to cl...
- 1976 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @szymon_dybczak ,Yes have databricks premium account and admin as well
- 2 kudos
- 547 Views
- 2 replies
- 2 kudos
Connectivity of dbx serverless to api gateway dns in aws is failing
Hello, I'm trying to establish the connection between dbx serverless account which is in "Account A" to the api gateway dns name which is in "Account B" is failing. i have created a api gateway with the custom domain for ex : "api-dev.amazon.com" and...
- 547 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @ilir_nuredini ,I'm trying the below way import requestsimport json API_URL = 'https://api-dev.amazon.com/v1/quality'headers = { "Authorization": f"Bearer {token}", "Content-Type": "application/json"}response = requests.get(API_URL, heade...
- 2 kudos
- 478 Views
- 1 replies
- 2 kudos
Error creating Databricks workspace on AWS (CREATE_FAILED with JSONDecodeError)
Hi everyone,I’m trying to create a Databricks workspace on AWS by following the official quick start guide:https://docs.databricks.com/aws/en/admin/workspace/quick-startHowever, my CloudFormation stack fails with the following error: The resource cr...
- 478 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Lovebo , I'm not too familiar with AWS but happy to have a go at helping. Let's see if we can rule out some of the more obvious things which are easy to miss on the documentation.First of all, are you just following the guide through: https://doc...
- 2 kudos
- 422 Views
- 2 replies
- 0 kudos
Configure job access_control_list block from a single place outside of job definition
Hi allIs it possible to configure the access_control_list block ((basically that contains permissions)) from a single place outside of Databricks job definition? This is changing as we are re-defining our permissions model, and each time resulting in...
- 422 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @noorbasha534 If you are trying to alter the permissions across many jobs at scale, I believe there are a couple of options to help speed up the process. Firstly, creating groups of users should help, as you can then easily change an entire gro...
- 0 kudos
- 503 Views
- 3 replies
- 0 kudos
databricks terraform provider, databricks_credential resource, service
I cannot make the databricks_credential resource create a service credential. It works fine with storage credentials. However, when i put `purpose = "SERVICE"` plus aws_iam_role and comment, in the apply phase it fails with `Error: cannot create cred...
- 503 Views
- 3 replies
- 0 kudos
- 0 kudos
I have the same error message now when trying to create a USE_SCHEMA grant for a service principal as in https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grant#schema-grants . I create a new service principal and th...
- 0 kudos
- 29317 Views
- 13 replies
- 12 kudos
Resolved! Installing libraries on job clusters
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
- 29317 Views
- 13 replies
- 12 kudos
- 12 kudos
Frankly, I'm also scratching my head about this library installation conundrum in job clusters; the missing Libraries tab is definitely throwing me off. Seems like init scripts are the only officially sanctioned route, which feels a bit clunky. Are t...
- 12 kudos
- 500 Views
- 3 replies
- 0 kudos
Databricks Execution Context scheduling
Databricks allows a cluster to have a maximum of 150 execution context. If a number of execution context is attached and running operations on a databricks cluster with a driver and n workers, then how is the execution scheduled? I am assuming that o...
- 500 Views
- 3 replies
- 0 kudos
- 0 kudos
Ok i get it.An execution context is not bound to cpu. It is like a session. So basically the limit of 150 execution contexts mean that 150 sessions/spark programs can run simultaneously on the cluster (whether that is possible on the hardware is an...
- 0 kudos
- 736 Views
- 2 replies
- 1 kudos
Resolved! AWS Account Level provider "databricks" Authentication
I am trying to deploy an additional workspace to my AWS Account with terraform, I keep running into this authentication an error message below, I have searched everywhere in my account console and used chatgpt which keep point me to the account con...
- 736 Views
- 2 replies
- 1 kudos
- 4709 Views
- 5 replies
- 6 kudos
Resolved! Issue with Databricks Alerts formatting sent to Microsoft Teams
Hello all,As of Thursday 20th 2025, we started to have our alert notifications being sent via Microsoft Teams with no formatting (raw HTML).We use a custom template with QUERY_RESULT_TABLE and a few html tags.Looks like databricks alerts deprecated ...
- 4709 Views
- 5 replies
- 6 kudos
- 6 kudos
Hello,Thanks for sharing the explanation and link to the Microsoft's documentation.We had removed the HTML formatting a few months ago.
- 6 kudos
- 1326 Views
- 5 replies
- 5 kudos
Resolved! Unable to Create Cluster in ADW Deployment — CONTROL_PLANE_REQUEST_FAILURE
I'm running into an issue with my Databricks workspace in Azure in my own VNet. I've successfully created two private endpoints: databricks_ui_api and browser_authenticationHowever, when I try to create a cluster, I get the following error:CONTROL_PL...
- 1326 Views
- 5 replies
- 5 kudos
- 5 kudos
Hello @eshwari Good day, I think szymon_dybczak and me provided enough information, please let me know if you had the solutions.If you found the solution useful, you can select the solution for the solution which helps others. Have a greate day!
- 5 kudos
- 1457 Views
- 3 replies
- 2 kudos
Resolved! GA for AIM
Do anyone know when Automatic Identity Management will be generally available for all the users?Announcing Automatic Identity Management for Azure Databricks | Databricks Blog
- 1457 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks for the clarification!I manage a fairly large Databricks environment, so enabling features that are still in Public Preview can be a bit challenging from a governance and stability standpoint. If possible, could you share any tentative timelin...
- 2 kudos
