- 5718 Views
- 7 replies
- 0 kudos
Failing PowerBi connection with data Bricks via SQL warehouse
I'm encountering an 'Invalid credentials' error (Session ID: 4601-b5a6-0daf792752a2, Region: us) when connecting Power BI to an Azure Databricks SQL Warehouse using an SPN. The SPN has CAN MANAGE access at SQL warehouse, admin rights at account and...
- 5718 Views
- 7 replies
- 0 kudos
- 0 kudos
Hello sai, sorry for the experience I usually available at desk during EMEA time zone. Apologies for the delay. Can you please try this?In "Advanced settings" of the data source within the gateway configuration and set the "Connection Encryption se...
- 0 kudos
- 2785 Views
- 1 replies
- 0 kudos
Looking for insights on enabling Databricks Automatic Provisioning
We currently have a SCIM provisioning connector set up to synchronize identities from Entra ID to Unity Catalog.We’re now considering enabling Databricks Automatic Provisioning but want to fully understand the potential impact on our environment befo...
- 2785 Views
- 1 replies
- 0 kudos
- 0 kudos
TranslateHello Xaveri.Good day!Here are few links related to databricks provisioninghttps://docs.databricks.com/aws/en/admin/users-groups/scim/aadhttps://www.databricks.com/blog/announcing-automatic-identity-management-azure-databricksBut do let me k...
- 0 kudos
- 4935 Views
- 4 replies
- 4 kudos
Privileged Identity Management for Databricks with Microsoft Entra ID
Privileged Identity Management (PIM) can be used to secure access to critical Databricks roles with Just-in-Time (JIT) access. This approach helps organizations enforce time-bound permissions, approval workflows, and centralized auditing for sensitiv...
- 4935 Views
- 4 replies
- 4 kudos
- 2534 Views
- 2 replies
- 0 kudos
Delete unassigned catalogs
Hi everybody,due to some not-so-optimal Infrastructure as code experiments with terraform I ended up a lot (triple digit) of catalogs in a metastore that are not assigned to any workspace and that i want to delete.Unfortunately, there is no way to ev...
- 2534 Views
- 2 replies
- 0 kudos
- 0 kudos
Yeah, I see those catalogs and i know that I could reattach and delete them. As i have around 100 those catalogs it would be nice to iterate through them by getting a list, e.g. using the cli or the rest API. And then force delete them , as described...
- 0 kudos
- 1852 Views
- 1 replies
- 1 kudos
VS Code - ipynb vs py execution - spark issue
Databricks Connect works inside VS Code notebook but the same code fails in a standalone script withValueError: default auth: cannot configure default credentialsI’m developing locally with **Databricks Connect 16.1.6** and VS Code.Inside a Jupyter n...
- 1852 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Sisi ,I think what's happening here is when you debug with option "Debug current file with Databricks Connect" then VS Code is using Databricks extension, which automatically handles authentication and sets up proper configuration.The regular Pyt...
- 1 kudos
- 4461 Views
- 6 replies
- 2 kudos
Out of memory error when installing environment dependencies of UC Python UDF
Hi,I've created a small UC Python UDF to test whether it works with custom dependencies (new PP feature), and every time I'm getting OOM errors with this message: [UDF_ENVIRONMENT_USER_ERROR.OUT_OF_MEMORY] Failed to install UDF dependencies for <cata...
- 4461 Views
- 6 replies
- 2 kudos
- 2 kudos
I tried with cluster, spent some couple of hours to load some libraries but unable to do. may be someone else can help you on this.
- 2 kudos
- 3565 Views
- 5 replies
- 4 kudos
Resolved! Metastore deletion issues
Good afternoon, I have an issue with my metastore in North Europe.All my workspaces got detached:If I go to Databricks console, I can see the metastore in North Europe I created.However, when I select the metastore in North Europe, I get the followin...
- 3565 Views
- 5 replies
- 4 kudos
- 4 kudos
I solved the issue by deleting all the asignments before deleting the metastore.1. Access to Databricks CLI and authenticate 2. List metastores>> databricks account metastores list 3. List wotrkspaces and check assignments>> databricks account worksp...
- 4 kudos
- 3174 Views
- 1 replies
- 1 kudos
Drop schema or catalog using cascade function
Hello In Databricks (non-Unity Catalog), I have two schemas (schema_a and schema_b) that both use the same root location in DBFS or external storage like ADLS.Example:abfss://container@storage_account.dfs.core.windows.net/data/project/schema_aabfss:/...
- 3174 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @EjB For the given example, here is the response:Will DROP SCHEMA schema_a CASCADE remove or affect tables in schema_b?No, unless:1. The tables in schema_a are managed tables, AND2. Tables in schema_b store their data physically inside /schema_...
- 1 kudos
- 3025 Views
- 3 replies
- 0 kudos
[INTERNAL_ERROR] Query could not be scheduled: HTTP Response code: 503. Please try again later
We have a databricks job configured to run a dbt project. The dbt cli compute cluster being used is serverless with a serverless sql warehouse. We encountered this error during a run. SQLSTATE: XX000Any idea why this occurred?
- 3025 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,We are experiencing same recurring HTTP errors throughout the day when executing queries on Databricks. The specific error message we receive is: "[INTERNAL_ERROR] Query could not be scheduled: HTTP Response code: 503. Please try again later. SQLS...
- 0 kudos
- 3024 Views
- 2 replies
- 3 kudos
Resolved! System tables performance optimization
Hi AllAre there any Databricks lab projects or GitHub repositories that leverage system tables to provide dashboards or code for monitoring, and more importantly, for optimizing workflows and clusters based on usage?
- 3024 Views
- 2 replies
- 3 kudos
- 3 kudos
+1 to @szymon_dybczak. I would also add the dashboards for DB SQL Warehouse monitoring.
- 3 kudos
- 4517 Views
- 6 replies
- 0 kudos
Unable to see sample data in Hive Metastore after moving to GCE
Hi,We have recently moved from GKE to GCE, it is taking forever to load the sample data in the manged delta tables.Even running simple select sql statements are taking forever. Totally clueless here, any help will be appreciatedThanks
- 4517 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi All,Strangely after struggle for 2 days we figured out that we can't run the cluster in scalable mode, so after selecting single node mode we are able to execute queries and job. It seems there is a bug in the Databrick's GKE to GCE migration. Won...
- 0 kudos
- 1833 Views
- 1 replies
- 0 kudos
Resolved! Best Approach to Retrieve Policy IDs Across Multiple Workspaces
Hi, I’m aware of the API endpoint api/2.0/policies/clusters/list to fetch a list of policy IDs and names. However, we have 50 different workspaces, and I need to retrieve the specific policy ID and name for each one.Could you advise on the most effic...
- 1833 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Chinu ,Databricks does not provide a global API to query all workspaces in a single call. I guess your only option for now is to use scripting approach.
- 0 kudos
- 10169 Views
- 2 replies
- 1 kudos
Resolved! [Azure Databricks]: Use managed identity to access mlflow models and artifacts
Hello! I am new to Azure Databricks and have a question: In my current setup, I am running some containerized python code within an azure functions app. In this code, I need to download some models and artifacts stored via mlflow in our Azure Databri...
- 10169 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @quad_t, were you able to find a solution to this problem? I'm having similar issues when trying to use MSI to connect to MLflow.
- 1 kudos
- 2900 Views
- 1 replies
- 1 kudos
Resolved! ADLS Gen2 with Unity Catalog on Azure Databricks / is Workspace Admin permissions sufficient?
Hello,I want to use an ADLS Gen2 Storage Account for Managed Delta Tables on Azure Databricks. The mounting/connection should be managed by Unity Catalog. There is only going to be a single workspace (for now).Does this require Account Admin permissi...
- 2900 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @raffael ,It depends on where you want to configure managed storage location. If you want to do this at Metastore level then you have to be an account admin and you need to do this during metastore creation.You can also configure storage location ...
- 1 kudos
- 3323 Views
- 2 replies
- 0 kudos
Resolved! Webhook Authentication
If want to send notifications via webhook to Splunk, Datadog, or LogicMonitor, how might I configure Databricks to authenticate using the destination platform's bearer token?
- 3323 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @prodrick ,It looks like webhooks destination supports only basic authentication using username and password. But you can try to paste bearer token to password section. Some webhook endpoints accept Bearer tokens in the password field while leavin...
- 0 kudos
