cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

danielrodri
by New Contributor II
  • 202 Views
  • 5 replies
  • 1 kudos

Resolved! Payment receipts of Databricks payments

Hello experts,I am trying to get receipts for the monthly payments done to Databricks. I need them for the financial department of the organization I am working for. The only billing information I get access to is the usage dashboards and the tables ...

  • 202 Views
  • 5 replies
  • 1 kudos
Latest Reply
danielrodri
New Contributor II
  • 1 kudos

Problem solved, thank you very much!

  • 1 kudos
4 More Replies
AndySkinner
by New Contributor III
  • 769 Views
  • 1 replies
  • 1 kudos

Troubleshooting Cluster

 We had a failure on a previously running fact table load (our biggest one) and it looked like an executor was failing due to a timeout error. As a test we upped the cluster size and changed the spark.executor.heartbeatinterval to 300s and the spark....

  • 769 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ismael-K
Databricks Employee
  • 1 kudos

The XXKDA error code is a general indicator for task scheduler issues or SPARK_JOB_CANCELLED.

  • 1 kudos
RakeshRoy
by New Contributor
  • 67 Views
  • 1 replies
  • 0 kudos

Community Accounts Getting Deleted Automatically

This is the second time in the same week, the accounts are getting vanished without any prior information. All the saved work has no backup I guess and no one would take responsibility because it a community version.Not a good thing to happen twice, ...

Error.png
  • 67 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

I would suggest to follow steps in https://community.databricks.com/t5/support-faqs/databricks-community-sso-august-3rd-2024/ta-p/78459 , if issue persist please contact databricks-community@databricks.com 

  • 0 kudos
sdheepak
by New Contributor II
  • 68 Views
  • 1 replies
  • 0 kudos

Databricks SQL Warehouse Hung - Queries Stuck in Queued State & No Alerts Triggered

We have been facing critical challenges with Databricks SQL Warehouse for the last four weeks. We are using Databricks SQL Warehouse injection from IICS, and we have observed the following issues:SQL Warehouse Going into a Hung State – The SQL Wareho...

  • 68 Views
  • 1 replies
  • 0 kudos
Latest Reply
Isi
New Contributor II
  • 0 kudos

Hey @sdheepak The first thing you need to identify is the type of SQL Warehouse you are using in Databricks:• Is it Serverless? If so, it is fully managed by Databricks, and you must contact Databricks support because you won’t have access to logs in...

  • 0 kudos
akmukherjee
by New Contributor III
  • 928 Views
  • 21 replies
  • 0 kudos

Resolved! Unable to enable Serverless Notebooks

Hello there,I have a Databricks Premium Subscription but am not able to enable Serverless Notebooks (as that option does not seem to exist). I have gone through DB documentation and have Unity Catalog Enabled. I even opened a ticket (00591635) but it...

  • 928 Views
  • 21 replies
  • 0 kudos
Latest Reply
Elderly
New Contributor II
  • 0 kudos

I'm also experiencing this issue:- I'm using a Premium Databricks subscription deployed on Azure in January 2025.- I've confirmed that Unity Catalogue is enabled in my subscription.- I've accessed my Account Console. Serveless Compute doesn't present...

  • 0 kudos
20 More Replies
dbuserng
by New Contributor
  • 96 Views
  • 1 replies
  • 0 kudos

JVM Heap Memory Graph - more memory used than available

I'm analyzing the memory usage of my Spark application and I see something strange when checking JVM Heap Memory Graph (see screenshot below). Each line on the graph is representing one executor.Why the memory usage sometimes reaches over 10GB, when ...

  • 96 Views
  • 1 replies
  • 0 kudos
Latest Reply
Sidhant07
Databricks Employee
  • 0 kudos

Hi @dbuserng , The memory usage in your Spark application can exceed the spark.executor.memory setting of 7GB for several reasons: • Off-Heap Memory Usage: Spark allows for off-heap memory allocation, which is not managed by the JVM garbage collector...

  • 0 kudos
Veilraj
by New Contributor II
  • 352 Views
  • 2 replies
  • 0 kudos

Configuration of NCC for Serverless to access SQL server running in a Azure VM

Hi Team, I am following this link to configure NCC for a Serverless compute to access a SQL Server running in a Azure VM. https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/This references to adding privat...

  • 352 Views
  • 2 replies
  • 0 kudos
Latest Reply
shhhhhh
New Contributor III
  • 0 kudos

also interested in doing this.  Have federated queries for Classic Databricks cluster pointing to SQL server, but can't find documentation for Serverless plane connecting to SQL server on a VM

  • 0 kudos
1 More Replies
Lawro
by New Contributor II
  • 546 Views
  • 5 replies
  • 0 kudos

Cannot list Clusters using Rest API

I am trying to run the following rest API command from:curl -H "Authorization: Bearer <PAT Code>" -X GET "curl -H "Authorization: Bearer <PAT Code>" -X GET "http://<databricks_workspace>.azuredatabricks.net/api/2.0/clusters/list"  When I run the comm...

  • 546 Views
  • 5 replies
  • 0 kudos
Latest Reply
Lawro
New Contributor II
  • 0 kudos

Hi, I definitely think it is facing network issues.  Its just very difficult to identify, when I am able to successfully ping the instance from the server originating the request.It is something jdbc related, just not sure what.it is

  • 0 kudos
4 More Replies
Rob_Lemmens
by New Contributor III
  • 774 Views
  • 9 replies
  • 1 kudos

Resolved! OAUTH Secrets Rotation for Service Principal through Databricks CLI

I am currently utilizing a specific Service Principal in my DevOps steps to utilize the Databricks CLI. It's using the OAuth tokens with M2M authentication (Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M) - Az...

  • 774 Views
  • 9 replies
  • 1 kudos
Latest Reply
Rob_Lemmens
New Contributor III
  • 1 kudos

 After filing a Microsoft Support Ticket through my client they provided me with the solution to the inquiry. There seems to be a undocumented API call that you can do to create this SP Oauth Client Secret and it works perfectly:curl -X POST --header...

  • 1 kudos
8 More Replies
NelsonE
by New Contributor III
  • 365 Views
  • 5 replies
  • 0 kudos

Resolved! Updating Workspace Cluster

Hello,My organization is experiencing difficulties updating our Google Kubernetes Engine (GKE) cluster.We've reviewed the official GKE documentation for automated cluster updates, but it appears to primarily focus on AWS integrations. We haven't foun...

  • 365 Views
  • 5 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

I apologize for the delay I am still looking into this

  • 0 kudos
4 More Replies
Behwar
by New Contributor II
  • 281 Views
  • 2 replies
  • 0 kudos

Databricks App in Azure Databricks with private link cluster (no Public IP)

Hello,I've deployed Azure Databricks with a standard Private Link setup (no public IP). Everything works as expected—I can log in via the private/internal network, create clusters, and manage workloads without any issues.When I create a Databricks Ap...

  • 281 Views
  • 2 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @Behwar, Did you make sure that your internal DNS is configured to map the web application workspace URL to your front-end VPC endpoint. This involves creating an A-record in your internal DNS that maps the workspace URL directly to the front-e...

  • 0 kudos
1 More Replies
arun_prakash
by New Contributor
  • 158 Views
  • 1 replies
  • 0 kudos

Databricks Apps not working in postman

I have a question regarding Databricks Apps. I have deployed my databricks Apps, and its working on my laptop, but when I try to open the same url in my mobile its redirecting to databricks signin page, and also its not working through postman as wel...

  • 158 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Is this issue specifically with the Databricks Apps am I right? Are you getting and erro message?

  • 0 kudos
andre-h
by New Contributor III
  • 490 Views
  • 4 replies
  • 3 kudos

Resolved! Install Python dependency on job cluster from a privately hosted GitLab repository (HTTPS/SSH)

Hello,We intend to deploy a Databricks workflow based on a Python wheel file which needs to run on a job cluster. There is a dependency declared in pyproject.toml which is another Python project living in a private Gitlab repository. We therefore nee...

  • 490 Views
  • 4 replies
  • 3 kudos
Latest Reply
hari-prasad
Valued Contributor II
  • 3 kudos

Hi @andre-h ,As a good alternative you can build the python package (wheel or egg) in your gitlab or github workflows and upload the package to dedicated cloud storage bucket. Then followed by you can specify the cloud storage path of your python lib...

  • 3 kudos
3 More Replies
drumcircle
by New Contributor II
  • 258 Views
  • 1 replies
  • 1 kudos

Determining spill from system tables

I'm trying to optimize machine selection (D, E, or L types on Azure) for job clusters and all-purpose compute and am struggling to identify where performance is sagging on account of disk spill.  Disk spill would suggest that more memory is needed.  ...

  • 258 Views
  • 1 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

For historical diagnostics, you might need to consider setting up a custom logging mechanism that captures these metrics over time and stores them in a persistent storage solution, such as a database or a logging service. This way, you can query and ...

  • 1 kudos
ambigus9
by New Contributor III
  • 602 Views
  • 8 replies
  • 3 kudos

PrivateLink Validation Error - When trying to access to Workspace

We have a workspace that had been deployed on AWS customer architecture using Terraform privatelink: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/aws-private-link-workspaceThe fact is when we disable the Public Acc...

ambigus9_0-1732035784493.png ambigus9_1-1732035847145.png ambigus9_2-1732037994364.png ambigus9_3-1732038098998.png
  • 602 Views
  • 8 replies
  • 3 kudos
Latest Reply
Walter_C
Databricks Employee
  • 3 kudos

Can you share your workspace id so I can do a validation?  

  • 3 kudos
7 More Replies
Labels