cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

ta
by New Contributor II
  • 8215 Views
  • 7 replies
  • 0 kudos

Can't access to manage account console

Hello!I have got problem with logging into manage account console.After clicking the Manage Account button, it takes me to the Workspace selection window and after selecting any of them, my workspace opens again.The first login was done by Acitve Dir...

  • 8215 Views
  • 7 replies
  • 0 kudos
Latest Reply
wellington
New Contributor III
  • 0 kudos

I'm facing the same problem, I´m admin on my workspace. Please someone from the Databricks team could respond what is happening when users can't access Accounts to create catalog ???????  It stay in infinite loops requesting the workspace name.

  • 0 kudos
6 More Replies
bfrank1972
by New Contributor III
  • 3612 Views
  • 3 replies
  • 0 kudos

Resolved! Small files and discrepancy in S3 vs catalog

Hello all,I'm in the process of optimizing my tables and I'm running into a confusing situation. I have a table named "trace_messages_fg_streaming_event". If I navigate to the Databricks catalog, it shows stats:Size: 6.7GB, Files: 464But when I look ...

bfrank1972_0-1697577884569.png
  • 3612 Views
  • 3 replies
  • 0 kudos
Latest Reply
bfrank1972
New Contributor III
  • 0 kudos

Nevermind, I'm dumb, figured it out, high frequency transactions leave many little files that need to be vacuumed. 

  • 0 kudos
2 More Replies
gilShin
by Contributor
  • 4624 Views
  • 5 replies
  • 2 kudos

Cannot access admin group members

Hi,I'm using Azure databricks premium tier and lately, when I'm creating a new workspace and after that trying to browse to the admin members page, I get the following error:Uncaught Error: No QueryClient set, use QueryClientProvider to set oneReload...

  • 4624 Views
  • 5 replies
  • 2 kudos
Latest Reply
KateGrim
New Contributor II
  • 2 kudos

Same issue here - both Chrome and Firefox.

  • 2 kudos
4 More Replies
gilShin
by Contributor
  • 3862 Views
  • 2 replies
  • 1 kudos

Resolved! Azure databricks issue

Hi,I'm using Azure databricks service and having trouble blocking public access to my workspace.The security groups that are attached to the subnets that databrics is using are being ignored.From what I've learned in the last couple of days, the data...

  • 3862 Views
  • 2 replies
  • 1 kudos
Latest Reply
gilShin
Contributor
  • 1 kudos

HiForgot to mention that I was using standard edition when writing this question but after struggling too long with that we've decided to upgrade to premium and use the IP access list feature and that worked!ThanksGil

  • 1 kudos
1 More Replies
viniaperes
by New Contributor II
  • 1361 Views
  • 0 replies
  • 0 kudos

How can I find my Account Executive?

I opened a support ticket to Databricks asking how I could whitelist an email domain to access Databricks Academy free self-paced training. I was told that I should see my account executive. How can I find it? Could anyone help me with this?

  • 1361 Views
  • 0 replies
  • 0 kudos
DX
by New Contributor II
  • 2692 Views
  • 2 replies
  • 0 kudos

Validation error on cluster_name is empty

Hi,When trying to update the Worker Type for a cluster that is specific for a job, I get a validation error saying the cluster name is empty.The name is present in the UI.In json view, the attribute is set to empty string ("")Swapping for a new clust...

  • 2692 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@DX Is that job cluster/all purpose cluster that you are using for Jobs 

  • 0 kudos
1 More Replies
OvZ
by New Contributor III
  • 1856 Views
  • 2 replies
  • 1 kudos

restrict ip access

Hi guru'sI working on Databricks azure. I have a question is it possible to restrict access (to the workspace url) based on ipadress other than  "enableIpAccessLists": "true"  in Databricks self ?   for example NSG/firewall/conditional access .. ?Tha...

  • 1856 Views
  • 2 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@OvZ if you are not going to provide access to user either resource level from azure they can not access databricks workspace right, usually recommended approach is to use access list that was suggested by @Retired_mod 

  • 1 kudos
1 More Replies
m997al
by Contributor III
  • 10795 Views
  • 9 replies
  • 5 kudos

Resolved! Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

Hi,We are attempting to set up Databricks with Unity Catalog (metastore) using a service principal (as opposed to the managed identity).Instructions we are using are here:  Create a Unity Catalog metastore - Azure Databricks | Microsoft LearnThe chal...

  • 10795 Views
  • 9 replies
  • 5 kudos
Latest Reply
m997al
Contributor III
  • 5 kudos

Thanks to all for the suggestions.  Ultimately, we went with the Managed Identity configuration (after all that investigation).  Answers very much appreciated.  Thank you.

  • 5 kudos
8 More Replies
StephanieAlba
by Databricks Employee
  • 6507 Views
  • 2 replies
  • 0 kudos

Resolved! What is best practice for determining how many workspaces I need?

What is the best practice for determining how many workspaces our company needs?  What are the most appropriate boundaries now that we have UC?

  • 6507 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@StephanieAlba adding to @Walter_C no of workspaces will depend on design that you are planning1. you can based on business unit2. you can just go by Dev/Qa/Tst/Prd for CICD Practices wise3. if you go based on above approach, UC schema/catalog wise w...

  • 0 kudos
1 More Replies
SivaPras_50542
by New Contributor
  • 1740 Views
  • 1 replies
  • 0 kudos

Tags are not getting added to the nodes

Tags which were added are not getting added to the EC2 instances,

SivaPras_50542_0-1697399714834.png
  • 1740 Views
  • 1 replies
  • 0 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 0 kudos

@SivaPras_50542 adding to @Retired_mod are you using clusters from pools, then the behavior will be different . tags wont get inherited from clusters for pools, we need to create tags to workspace level/pool level, please check below linkhttps://docs...

  • 0 kudos
StephanieAlba
by Databricks Employee
  • 4180 Views
  • 3 replies
  • 1 kudos

Resolved! How many workspaces should we have?

There is a default limitation for workspace numbers in a Databricks account, which is 50 for an enterprise account: https://docs.databricks.com/en/resources/limits.html What is the best practice if we need more than 50 workspaces? Will we need more a...

Administration & Architecture
Unity Catalog
Workspaces
  • 4180 Views
  • 3 replies
  • 1 kudos
Latest Reply
Raluka
New Contributor III
  • 1 kudos

Thank you so much for helping me. I didn't even expect it.

  • 1 kudos
2 More Replies
Masoomeh
by New Contributor II
  • 1845 Views
  • 0 replies
  • 0 kudos

Problem with provisioning lakehouse via Terraform

I am trying to provision the lakehouse on Azure using the Terraform template provided here: https://github.com/databricks/terraform-databricks-examples/tree/main/examples/adb-lakehouse. I have started from scratch and have only added a resource group...

  • 1845 Views
  • 0 replies
  • 0 kudos
JoeStringer
by New Contributor II
  • 1749 Views
  • 1 replies
  • 1 kudos

Help Needed: Grants remaining for removed Groups and Service Principal

We have an issue in a Workspace which is managed by Terraform, a change went in to update the Group and Service Principal (SP) names but due to the internal ordering the Groups and SP were removed and replaced before the Grants were updated.If we now...

  • 1749 Views
  • 1 replies
  • 1 kudos
Latest Reply
YennickT
New Contributor II
  • 1 kudos

We are experiencing a similar issue, except with the storage credential resource. We created some storage credentials using Terraform, but when trying to destroy them using Terraform they were ignored. So we decided to manually delete the storage cre...

  • 1 kudos
HappySK
by New Contributor
  • 2026 Views
  • 1 replies
  • 1 kudos

Why do we need different AWS Accounts ?

I am just going through the AWS Databricks Platform Administration course and I am curious to know about the Cloud Accounts used to setup DatabricksWhen we are setting up the Databricks, we are using AWS Account but we are not using that account sayi...

  • 2026 Views
  • 1 replies
  • 1 kudos
Latest Reply
dleblanc
Databricks Employee
  • 1 kudos

Hi @HappySK,  The AWS account you're using basically serves as the data plane - this is where your data lives, and the compute resources that process it (at least in the classical compute model) will be provisioned there as well. As part of the confi...

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels