cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

knawara
by Contributor
  • 4486 Views
  • 1 replies
  • 0 kudos

Databricks Bundles - Terraform state management

Hello,I had a look at DABs today and it seems they are using Terraform under the hood. The state is stored in Databricks Workspace, in the bundle deployment directory. Is it possible to use just the state management functionality that DABs must have ...

  • 4486 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ismael-K
Databricks Employee
  • 0 kudos

While it may be possible to use the state management functionality provided by DABs using Terraform, it would require additional effort to synchronize your code and manage state consistently. The choice would depend on the use case and we should keep...

  • 0 kudos
--V
by New Contributor II
  • 1715 Views
  • 3 replies
  • 3 kudos

How to tag databricks workspace to keep track of AWS resources.

How to add tags to a databricks workspace so that tag propagates to all cloud resources created for or by workspace to keep track of their costs.

  • 1715 Views
  • 3 replies
  • 3 kudos
Latest Reply
Edward357J
New Contributor III
  • 3 kudos

Hello!To add tags to a Databricks workspace and ensure they propagate to all cloud resources created for or by the workspace, log in to your Databricks workspace and navigate to the workspace settings or administration console. Use the tag interface ...

  • 3 kudos
2 More Replies
ptco
by New Contributor II
  • 6678 Views
  • 11 replies
  • 3 kudos

"Azure Container Does Not Exist" when cloning repositories in Azure Databricks

Good Morning, I need some help with the following issue:I created a new Azure Databricks resource using the vnet-injection procedure. (here) I then proceeded to link my Azure Devops account using a personal account token. If I try to clone a reposito...

  • 6678 Views
  • 11 replies
  • 3 kudos
Latest Reply
NandiniN
Databricks Employee
  • 3 kudos

Yes, the container is indeed the container in the storage account deployed by the Databricks instance into the managed resource group. This storage account is part of the managed resource group associated with your Azure Databricks workspace. If the ...

  • 3 kudos
10 More Replies
chenda
by New Contributor II
  • 7186 Views
  • 1 replies
  • 3 kudos

Databricks Cluster Failed to Start - ADD_NODES_FAILED (Solution)

Lately we encountered the issue that the classic compute clusters could not start. With the help of Databricks team to troubleshoot, we found the issue and get it fixed. So I think writing it here could help other people who would encounter the same ...

chenda_0-1728285163027.png chenda_1-1728285255900.png chenda_2-1728285850870.png chenda_8-1728289622254.png
  • 7186 Views
  • 1 replies
  • 3 kudos
Latest Reply
jem
New Contributor III
  • 3 kudos

Thanks for sharing. We had the same problem. I missed to add private endpoints to the workspace storage account in the managed resource group. I will also add NCC rules in the Databricks account. Then you don't need the subnets in the firewall.

  • 3 kudos
markbaas
by New Contributor III
  • 1637 Views
  • 3 replies
  • 3 kudos

private endpoint to non-storage azure resource

I'm trying to set up a ncc and private endpoint for a container app environment in azure. However I get the following error:Error occurred when creating private endpoint rule: : BAD_REQUEST: Can not create Private Link Endpoint with name databricks-x...

  • 1637 Views
  • 3 replies
  • 3 kudos
Latest Reply
markbaas
New Contributor III
  • 3 kudos

All the azure subscriptions have this registered. Could this not be a azure subscription within the databricks tenant?

  • 3 kudos
2 More Replies
jyunnko
by New Contributor
  • 701 Views
  • 1 replies
  • 1 kudos

How to find the billing of each cell in a notebook?

Suppose I have run ten different statements/tasks/cells in a notebook, and I want to know how many DBUs each of these ten tasks used. Is this possible?

  • 701 Views
  • 1 replies
  • 1 kudos
Latest Reply
Isi
Honored Contributor III
  • 1 kudos

Hey,I really think this it’s not possible to directly determine the cost of a single cell in Databricks.However, you can approach this in two ways, depending on the type of cluster you’re using, as different cluster types have different pricing model...

  • 1 kudos
jaytimbadia
by New Contributor II
  • 673 Views
  • 3 replies
  • 0 kudos

GPU accelerator not matching with desired memory.

Hello, We have opted for Standard_NC8as_T4_v3 which claims to have 56GB memory. But, when I am doing nvidia-smi in the notebook, its showing only ~16 GB, Why?Please let me know what is happening here?  Jay

  • 673 Views
  • 3 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Please refer to: https://learn.microsoft.com/en-us/azure/databricks/compute/gpu

  • 0 kudos
2 More Replies
Hari-dbw
by New Contributor
  • 460 Views
  • 0 replies
  • 0 kudos

Error "Gateway authentication failed for 'Microsoft.Network'" While Creating Azure Databricks

Hi All,I'm encountering an issue while trying to create a Databricks service in Azure. During the setup process, I get the following error:"Gateway authentication failed for 'Microsoft.Network'"I've checked the basic configurations, but I'm not sure ...

  • 460 Views
  • 0 replies
  • 0 kudos
erigaud
by Honored Contributor
  • 2029 Views
  • 5 replies
  • 5 kudos

Resolved! Databricks cluster pool deployed through Terraform does not have UC enabled

Hello everyone,we have a workspace with UC enabled, we already have a couple of catalogs attached and when using our personal compute we are able to read/write tables in those catalogs.However for our jobs we deployed a cluster pool using Terraform b...

erigaud_1-1736874136257.png
  • 2029 Views
  • 5 replies
  • 5 kudos
Latest Reply
erigaud
Honored Contributor
  • 5 kudos

Confirmed that this works ! THank you 

  • 5 kudos
4 More Replies
karthiknuvepro
by New Contributor II
  • 744 Views
  • 1 replies
  • 0 kudos

GCP Databricks | Workspace Creation Error: Storage Credentials Limit Reached

Hi Team,We are encountering an issue while trying to create a Databricks Workspace in the GCP region us-central1. Below is the error message:Error Message:Workspace Status: FailedDetails: Workspace failed to launch.Error: BAD REQUEST: Cannot create 1...

  • 744 Views
  • 1 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @karthiknuvepro, Do you have an active support plan? Over a ticket with us we can request the increase of this limit.

  • 0 kudos
AnkitShah
by New Contributor II
  • 1772 Views
  • 4 replies
  • 0 kudos

How do we get user list who accessed/downloaded specific model in Unity catalog for last 6 months

How do we get user list who accessed/downloaded specific model in Unity catalog for last 6 months

  • 1772 Views
  • 4 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @AnkitShah, I just tried on my end and found these 2 tables that might be useful. They do not exact show who downloaded a model artifact but who interacted with it: https://docs.databricks.com/en/ai-gateway/configure-ai-gateway-endpoints.html#usag...

  • 0 kudos
3 More Replies
RicksDB
by Contributor III
  • 1969 Views
  • 6 replies
  • 1 kudos

Governance to restrict compute creation

Hi,Cluster policies used to be an easy way to handle governance on computes. However, more and more, there seem to be no way to control many new compute features within the platform. We currently have this issue for model serving endpoints and vector...

  • 1969 Views
  • 6 replies
  • 1 kudos
Latest Reply
nskiran
New Contributor III
  • 1 kudos

If you are looking to restrict end users to create certain cluster configuration only, you can do so by using databricks APIs. Through python and Databricks API, you can specify what kind of cluster configurations are allowed and also restrict users ...

  • 1 kudos
5 More Replies
dbuserng
by New Contributor II
  • 3234 Views
  • 1 replies
  • 4 kudos

High memory usage on Databricks cluster

In my team we have a very high memory usage even when the cluster has just been started and nothing has been run yet. Additionally, memory usage never drops to lower levels - total used memory always fluctuates around 14GB.Where is this memory usage ...

  • 3234 Views
  • 1 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

This is not necessarily an issue.  Linux uses a lot of RAM for caching but this does not mean it cannot be released for processes (dynamic memory mgmt).Basically the philosophy is that RAM that is not used (so actually 'free') is useless.Here is a re...

  • 4 kudos
jimbender
by New Contributor II
  • 1075 Views
  • 1 replies
  • 0 kudos

Newbie DAB question regarding wheels

I am trying to build a wheel using a DAB.  It errors saying I don't have permissions to install my wheel onto a cluster I am have been given.  Is it possible to just upload the wheel to a subdir the /Shared directory and use it from there instead of ...

  • 1075 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

May I know the exact error you are getting on the cluster?You can use the following code to use a wheel in a shared folder: resources: jobs: my-job: name: my-job tasks: - task_key: my-task new_cluster: ...

  • 0 kudos