cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Fresher
by New Contributor II
  • 2854 Views
  • 4 replies
  • 1 kudos

Databricks is taking too long to run a query

I ran a simple query. It is taking too much time and it doesn't stop running. I am the only one who is using cluster. It's happening to every notebook in the workspace.

  • 2854 Views
  • 4 replies
  • 1 kudos
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

Do you see spark jobs running in Spark UI ?

  • 1 kudos
3 More Replies
harripy
by New Contributor III
  • 2409 Views
  • 2 replies
  • 1 kudos

Resolved! Native service principal support in JDBC/ODBC drivers

Read from Databricks integration best practises about the native support for Service Principal authentication on JDBC/ODBC drivers. The timetable mentioned for this was "expected to land in 2023", is this referring to the https://docs.databricks.com/...

  • 2409 Views
  • 2 replies
  • 1 kudos
Latest Reply
daniel_sahal
Esteemed Contributor
  • 1 kudos

@harripy As it says in the documentation, JDBC driver 2.6.36 and above supports Azure Databricks OAuth secrets for OAuth M2M or OAuth 2.0 client credentials authentication. Microsoft Entra ID secrets are not supported.https://learn.microsoft.com/en-u...

  • 1 kudos
1 More Replies
Swoops
by New Contributor II
  • 1978 Views
  • 1 replies
  • 2 kudos

Resolved! What happens to the Workspace when deleting the user that created that workspace

Hi,Background:In Azure-Datebricks:We have a personal account that created a workspace, this workspace is our main workspace. The users Microsoft Office account is already deleted but the user still exist in Databricks.Question 1 is:What happens to th...

  • 1978 Views
  • 1 replies
  • 2 kudos
Latest Reply
arpit
Databricks Employee
  • 2 kudos

What happens to this workspace if we delete the User from workspace? Will the Workspace be deleted? → Workspace will not get deleted. It will continue to function.Is it possible to change Workspace own to another account i.e. a service account. If so...

  • 2 kudos
Dicer
by Valued Contributor
  • 2929 Views
  • 1 replies
  • 1 kudos

Resolved! Import folder (no .whl or .jar files) and run the `python3 setup.py bdist_wheel` for lib install

I want to import the ibapi python module in Azure Databricks Notebook.Before this, I downloaded the the TWS API folder from https://interactivebrokers.github.io/# I need to go through the following steps to install the API:Download and install TWS Ga...

  • 2929 Views
  • 1 replies
  • 1 kudos
Latest Reply
arpit
Databricks Employee
  • 1 kudos

You can try to upload the folder in the workspace location and try to cd in the desired folder and try to install in via notebook. But it would be a notebook scope installation. If you are looking for a cluster scoped installation then you would need...

  • 1 kudos
Datastoryteller
by New Contributor II
  • 1540 Views
  • 3 replies
  • 0 kudos

Permission denied using patchelf

Hello all I would like to run a Python script on a Shared cluster: The Python script tries to do, under the hood, a call to `patchelf` utility, in order to set a r-path. Something along those lines:execute('patchelf',['--set-rpath', rpath, lib])The p...

Screenshot 2024-04-17 at 16.00.03.png
  • 1540 Views
  • 3 replies
  • 0 kudos
Latest Reply
Datastoryteller
New Contributor II
  • 0 kudos

I basically followed this tutorial https://learn.microsoft.com/en-gb/azure/databricks/data-governance/unity-catalog/get-started and it seemed to be working, so I guess Unity Catalog is enabled?

  • 0 kudos
2 More Replies
Kim3
by New Contributor II
  • 1950 Views
  • 0 replies
  • 0 kudos

Delete Azure Databricks Workspace resource but reference remains in Account Console

As the title says. I have deleted both the Azure Databricks Workspace resource and the resource group it was located in. However, I can still see the workspace in the account console. Also, it appears in the response from GET https://accounts.azureda...

  • 1950 Views
  • 0 replies
  • 0 kudos
erigaud
by Honored Contributor
  • 4535 Views
  • 2 replies
  • 0 kudos

Resolved! Call an Azure Function App with Access Restrictions from a Databricks Workspace

Hello,As the title says, I am trying to call an function from an Azure Function App configured with access restrictions from a python notebook in my Databricks workspace. The Function App resource is in a different subscription as the Databricks work...

  • 4535 Views
  • 2 replies
  • 0 kudos
Latest Reply
erigaud
Honored Contributor
  • 0 kudos

Update : Problem was fixed ! The key was to set an VNET rule in the access restriction, giving access directly to the subnets used by Databricks.It seems like for Microsoft to Microsoft connections, the IP addresses are not used, so adding the IP ran...

  • 0 kudos
1 More Replies
alm
by New Contributor III
  • 1946 Views
  • 0 replies
  • 0 kudos

Azure Devops repos access

I have a Databricks setup, where the users and their permissions are handled in Microsoft Azure using AD groups and then provisioned (account level) using a provisioning connector to Databricks. The code repositories are in Azure Devops where users a...

  • 1946 Views
  • 0 replies
  • 0 kudos
sushant047_ms
by New Contributor III
  • 4753 Views
  • 3 replies
  • 2 kudos

How to bind a User assigned Managed identity to Databricks to access external resources?

Is there a way to bind a user assigned managed identity to Databricks? We want to access some SQL DBs, Redis cache from our Spark code running on Databricks using Managed Identity instead of Service Principals and basic authentication.As of today, Da...

Administration & Architecture
azure
managed-identity
user-assigned-managed-identity
  • 4753 Views
  • 3 replies
  • 2 kudos
Latest Reply
sushant047_ms
New Contributor III
  • 2 kudos

@Carpender correcting my comment above, Databricks assigned Managed Identity is working and we are able to access but as stated in the original question we are looking for authorization using User Assigned Managed Identity (UAMI). With UAMI we cannot...

  • 2 kudos
2 More Replies
DavidZS
by New Contributor
  • 5201 Views
  • 1 replies
  • 0 kudos

How to setup service principal to assing account-level groups to workspaces using terraform

Based on best practices, we have set up SCIM provisioning using Microsoft Entra ID to synchronize Entra ID groups to our Databricks account. All workspaces have identity federation enabled.However, how should workspace administrators assign account-l...

  • 5201 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Have you tried giving Manager role on the group to the service principal which is workspace admin? Once you do this you may be able to use the settings to  In workspace context, adding account-level group to a workspace in databricks_permission_assig...

  • 0 kudos
mroy
by Contributor
  • 1585 Views
  • 1 replies
  • 0 kudos

Resolved! Instances are not being terminated in time (extra AWS costs)

For a few days we have been trying to figure out why our AWS costs suddenly went up around March 20th, and we just found the answer: the EC2 instances are left in an unterminated state for a couple of minutes at the end of each run! This is a very se...

mroy_0-1712023586396.png
  • 1585 Views
  • 1 replies
  • 0 kudos
Latest Reply
mroy
Contributor
  • 0 kudos

Nevermind, this was actually due to a reservation that expired. 臘‍

  • 0 kudos
rocky5
by New Contributor III
  • 3286 Views
  • 5 replies
  • 0 kudos

System.billing.usage table - cannot match job_id from databricks api/UI

Hello, I have multiple continuous jobs that are running for many days (Kafka stream), however querying System.billing.usage table by job_id from UI or databricks job api not return any results for those jobs.1. What is the reason behind that?2. If I ...

  • 3286 Views
  • 5 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Hello, you are right, apologize for my misunderstanding as you have mentioned the job id persist, is the job run id the one that will be changed. So your issue is that as the job is continuing running the information of that run is not showing in the...

  • 0 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels