cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

akmukherjee
by New Contributor III
  • 1073 Views
  • 21 replies
  • 0 kudos

Resolved! Unable to enable Serverless Notebooks

Hello there,I have a Databricks Premium Subscription but am not able to enable Serverless Notebooks (as that option does not seem to exist). I have gone through DB documentation and have Unity Catalog Enabled. I even opened a ticket (00591635) but it...

  • 1073 Views
  • 21 replies
  • 0 kudos
Latest Reply
Elderly
New Contributor II
  • 0 kudos

I'm also experiencing this issue:- I'm using a Premium Databricks subscription deployed on Azure in January 2025.- I've confirmed that Unity Catalogue is enabled in my subscription.- I've accessed my Account Console. Serveless Compute doesn't present...

  • 0 kudos
20 More Replies
jx1226
by New Contributor III
  • 111 Views
  • 2 replies
  • 0 kudos

Cross-tenant networking two orgs on Azure

Hi Databricks Community,For one client, we are looking for best practices, steps to establish cross-tenant networking and landing zones between two organizations. The current situation is that the source systems are in one organization, while the Azu...

  • 111 Views
  • 2 replies
  • 0 kudos
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Cross posting https://community.databricks.com/t5/administration-architecture/databricks-azure-cross-tenant-connection-to-storage-account/td-p/82491#:~:text=I%20tested%20the%20connection%20to,when%20tested%20from%20this%20VM. 

  • 0 kudos
1 More Replies
MarcoRezende
by New Contributor II
  • 1737 Views
  • 3 replies
  • 4 kudos

It's possible to sync account group/user to workspace without the need to do it manually?

I am using Databricks SCIM for my Databricks Account, so when i add a user or group in the SCIM connector, the user or group its created in Databricks Account. After this, i need to manually assign the user/group to the workspaces. My boss wants to o...

  • 1737 Views
  • 3 replies
  • 4 kudos
Latest Reply
saurabh18cs
Valued Contributor III
  • 4 kudos

Hi , I agree with @Rjdudley EntraID groups are better 

  • 4 kudos
2 More Replies
Mihai_LBC
by New Contributor II
  • 206 Views
  • 2 replies
  • 1 kudos

Resolved! Table size information display

Hi,I have a problem with displaying information about the size of my tables.This information is visible several times, but after a while it disappears again.I need to undestand what is happen, and why this information is not available all the time on...

Information available.png Missing information.png
  • 206 Views
  • 2 replies
  • 1 kudos
Latest Reply
JakubSkibicki
Contributor
  • 1 kudos

In general you need a cluster or warehouse to be active for those detail to be presented.   

  • 1 kudos
1 More Replies
axelboursin
by New Contributor II
  • 119 Views
  • 0 replies
  • 0 kudos

Need to create an Identity Federation between my Databricks workspace/account and my AWS account

Hello,I need to set up an identification between my Databricks workspace/account and my AWS account, where Databricks is already deployed.The goal is to make an easy authentification without access and secret keys.So I thought that OIDC will be the s...

  • 119 Views
  • 0 replies
  • 0 kudos
dbuserng
by New Contributor II
  • 129 Views
  • 1 replies
  • 0 kudos

Driver: how much memory is actually available?

I have a cluster where Driver type is Standard_DS3_v2 (14GB Memory and 4 Cores). When I use free -h command in Web terminal (see attached screenshot) I get the response that I only have 8.9GB memory available on my driver - why is that?fyi, spark.dri...

  • 129 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi @dbuserng , The free -h command in the web terminal shows only 8.9GB of available memory on your driver, which is a Standard_DS3_v2 instance with 14GB of memory, because Databricks has services running on each node. This means the maximum allowabl...

  • 0 kudos
dbuserng
by New Contributor II
  • 112 Views
  • 1 replies
  • 0 kudos

JVM Heap Memory Graph - more memory used than available

I'm analyzing the memory usage of my Spark application and I see something strange when checking JVM Heap Memory Graph (see screenshot below). Each line on the graph is representing one executor.Why the memory usage sometimes reaches over 10GB, when ...

  • 112 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi @dbuserng , The memory usage in your Spark application can exceed the spark.executor.memory setting of 7GB for several reasons: • Off-Heap Memory Usage: Spark allows for off-heap memory allocation, which is not managed by the JVM garbage collector...

  • 0 kudos
sruthianki
by New Contributor II
  • 189 Views
  • 4 replies
  • 0 kudos

photon is being used by a job or not

We have lots of customers using many job as well as interactive clusters with photon enabled which is drastically increasing the cost .We would like to know if there is any table in system or any details that we can get through API that lists if the ...

  • 189 Views
  • 4 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi @sruthianki , If you want to check if the job is really using photon or not you can check the SQL query plan in spark UI for its stages and the metrics will highlighted in yellow colour.

  • 0 kudos
3 More Replies
Kroy
by Contributor
  • 18454 Views
  • 2 replies
  • 1 kudos

How to Know DBU consumption in azure databricks ?

In Azure portal - Billing we can get the COST but how to know How much DBU is consumed ?

  • 18454 Views
  • 2 replies
  • 1 kudos
Latest Reply
odoll
New Contributor II
  • 1 kudos

There was a promo on serverless early 2024 at the time which at some point got extended, and was bigger depending where you were.

  • 1 kudos
1 More Replies
ma10
by New Contributor
  • 346 Views
  • 1 replies
  • 0 kudos

Issue with updating email with SCIM Provisioning

Hi all,For our set-up we have configured SCIM provisioning using Entra ID, group assignment on Azure is dealt with by IdentityIQ Sailpoint, and have enabled SSO for Databricks. It has/is working fine apart from one scenario. The original email assign...

  • 346 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ismael-K
Databricks Employee
  • 0 kudos

Currently, the email address is an immutable attribute in the Databricks application. To request a change to this behavior, you can submit a feature enhancement. In the interim, you can also submit a support case for a potential workaround.

  • 0 kudos
faaiqsaheed12
by New Contributor II
  • 5354 Views
  • 11 replies
  • 4 kudos

Resolved! Unable to view Manage Account Option in Databricks UI

Hi All, I want to remove Unity Catalog from Admin Console so that I need to navigate to manage account option but the option is not available in my Databricks workspace. Please help me to sort this issue by removing Unity Catalog.

  • 5354 Views
  • 11 replies
  • 4 kudos
Latest Reply
Amit_Dass
New Contributor II
  • 4 kudos

Brilliant! I followed the above and its works seamlessly. 

  • 4 kudos
10 More Replies
yumnus
by New Contributor III
  • 131 Views
  • 3 replies
  • 0 kudos

Not able to connect to GCP Secret Manager except when using "No isolation shared" Cluster

Hey everyone,We’re trying to access secrets stored in GCP Secret Manager using its Python package from Databricks on GCP. However, we can only reach the Secret Manager when using "No Isolation Shared" clusters, which is not an option for us. Currentl...

  • 131 Views
  • 3 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @yumnus, Could you please share the full error trace? what is the package you are installing? One suggestion, instead of relying on the metadata server, you can use a service account key file to authenticate with GCP Secret Manager. You can sto...

  • 0 kudos
2 More Replies
Veilraj
by New Contributor II
  • 399 Views
  • 2 replies
  • 0 kudos

Configuration of NCC for Serverless to access SQL server running in a Azure VM

Hi Team, I am following this link to configure NCC for a Serverless compute to access a SQL Server running in a Azure VM. https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/This references to adding privat...

  • 399 Views
  • 2 replies
  • 0 kudos
Latest Reply
shhhhhh
New Contributor III
  • 0 kudos

also interested in doing this.  Have federated queries for Classic Databricks cluster pointing to SQL server, but can't find documentation for Serverless plane connecting to SQL server on a VM

  • 0 kudos
1 More Replies
Lawro
by New Contributor II
  • 625 Views
  • 5 replies
  • 0 kudos

Cannot list Clusters using Rest API

I am trying to run the following rest API command from:curl -H "Authorization: Bearer <PAT Code>" -X GET "curl -H "Authorization: Bearer <PAT Code>" -X GET "http://<databricks_workspace>.azuredatabricks.net/api/2.0/clusters/list"  When I run the comm...

  • 625 Views
  • 5 replies
  • 0 kudos
Latest Reply
Lawro
New Contributor II
  • 0 kudos

Hi, I definitely think it is facing network issues.  Its just very difficult to identify, when I am able to successfully ping the instance from the server originating the request.It is something jdbc related, just not sure what.it is

  • 0 kudos
4 More Replies
Lawro
by New Contributor II
  • 102 Views
  • 1 replies
  • 0 kudos

JDBC Connect Time out

Anyone know why I would get the JDBC Connect error below:java.sql.SQLException: [Databricks][JDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: com.databricks.client.jdbc42.internal.apache.http.conn.ConnectTimeoutExc...

  • 102 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @Lawro, That normally happens whenever there is a network issue or firewall blocking the request. Is it failing consistently and have you tested connectivity to your SQL instance using nc -vz command via a notebook?

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels