cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RonMoody
by New Contributor II
  • 2831 Views
  • 2 replies
  • 0 kudos

service principal table accesses not showing up in system.audit

When we run jobs using service principals system.audit doesn't show any table accesses (getTable). Volume (getVolume) shows up for service principals. Same query when run as a user shows up in system.audit. I know system.audit is in public preview. W...

  • 2831 Views
  • 2 replies
  • 0 kudos
Latest Reply
RonMoody
New Contributor II
  • 0 kudos

hi @Kaniz thanks so much for your reply ! I was referring to https://docs.databricks.com/en/administration-guide/system-tables/audit-logs.html which is part of databricks core offering and isn't related to ServiceNow's offering. I am assuming they ar...

  • 0 kudos
1 More Replies
cltj
by New Contributor III
  • 1053 Views
  • 2 replies
  • 1 kudos

Three level name space naming standard

Hi all, I have not been successful in getting a good grip of the naming conventions for the three level name space. Initially i learned about bronze, silver and gold, but i am confused where to put this. The obvious choice may be to use the {catalog}...

  • 1053 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @cltj ," I am also looking for others comments on my attached picture. " Could not find any.  

  • 1 kudos
1 More Replies
valjas
by New Contributor III
  • 1075 Views
  • 1 replies
  • 0 kudos

Is it possible to change the Azure storage account of Unity Catalog?

We have unity catalog metastore set up in storage account prod_1. Can we move this to prod_2 storage account and delete prod_1?Also, is it possible to rename the catalogs once they are created?

  • 1075 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @valjas , Certainly! Let’s address your questions:   Moving Unity Catalog Metastore: Yes, you can move the Unity Catalog metastore from one storage account (e.g., prod_1) to another (e.g., prod_2).Follow these steps: Create a new Premium perform...

  • 0 kudos
aayusha3
by New Contributor II
  • 1804 Views
  • 5 replies
  • 2 kudos

Internal error: Attach your notebook to a different compute or restart the current compute.

I am currently using a personal computer cluster [13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)] on GCP attached to a notebook. After running a few command lines without an issue, I end up getting this error  Internal error. Attach your notebook...

aayusha3_0-1700411861325.png
  • 1804 Views
  • 5 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @aayusha3 , @amandaolens , @Martin74 , The error message you encountered while using your personal computer cluster with Databricks Runtime 13.3 LTS (which includes Apache Spark 3.4.1 and Scala 2.12) on Google Cloud Platform (GCP) indicates an iss...

  • 2 kudos
4 More Replies
amoghjain
by New Contributor II
  • 747 Views
  • 1 replies
  • 0 kudos

Standard_NC8as_T4_v3" and "Standard_NC4as_T4_v3" instances

I am running into an issue where "Standard_NC8as_T4_v3" and "Standard_NC4as_T4_v3" instances are behaving differently for a 30gb custom docker image, and I am a bit stumped.when using NC4 instances, I get a timeout, with the exact message shown below...

  • 747 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @amoghjain, Certainly! It seems you’re encountering a timeout issue when pulling a custom Docker image on your NC4 instances, while the NC8 instances work fine.   Let’s explore some potential reasons and pointers:   Network Configuration: Check th...

  • 0 kudos
Learnit
by New Contributor II
  • 1412 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks deployment and automation tools comparison.

Hello All, As a newcomer to databricks, I am seeking guidance on automation within databricks environments. What are the best best practices for deployment, and how do Terraform, the REST API, and the databricks SDK compare in terms of advantages and...

  • 1412 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Learnit , Certainly! As a newcomer to Azure Databricks, understanding best practices for deployment and automation is crucial.    Let’s explore some recommendations and compare the tools you’ve mentioned:   Best Practices for Deployment in Azure ...

  • 0 kudos
amandaolens
by New Contributor III
  • 3453 Views
  • 6 replies
  • 2 kudos

Resolved! Internal error. Attach your notebook to a different compute or restart the current compute. java.lan

Internal error. Attach your notebook to a different compute or restart the current compute.java.lang.RuntimeException: abort: DriverClient destroyed at com.databricks.backend.daemon.driver.DriverClient.$anonfun$poll$3(DriverClient.scala:577) at scala...

  • 3453 Views
  • 6 replies
  • 2 kudos
Latest Reply
amandaolens
New Contributor III
  • 2 kudos

@Kaniz yeah, in one dataset there are slightly higher data points.schema are same.when the spark crashed i have checked the memory usage.it was around 50%.

  • 2 kudos
5 More Replies
lgepp11
by New Contributor II
  • 3016 Views
  • 2 replies
  • 1 kudos

Azure Entra SSO Error: Your user has not been registered

I have set up SSO within databricks and automatic user provisioning with Azure Entra and confirmed it is working for all users. However 1 user is presented with this when signing in. The user is in the enterprise app within Azure Entra and the user i...

lgepp11_0-1696914264539.png
Administration & Architecture
azure
Entra
Error
Sign In
sso
  • 3016 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @lgepp11 , This issue could be caused by a number of factors, such as incorrect configuration settings, network issues, or browser-related issues. To troubleshoot this issue, I recommend checking the following:   Ensure that the user is assigned t...

  • 1 kudos
1 More Replies
niklas
by Contributor
  • 1892 Views
  • 1 replies
  • 0 kudos

Resolved! Error: cannot create metastore data access

I'm in the progress of enabling Databricks Unity Catalog and encountered a problem with the databricks_metastore_data_access Terraform resource:resource "databricks_metastore_data_access" "this" { provider = databricks.account-level metastore_id...

Administration & Architecture
metastore
Terraform
Unity Catalog
  • 1892 Views
  • 1 replies
  • 0 kudos
Latest Reply
niklas
Contributor
  • 0 kudos

Found a solution. Please see my answer on Stack Overflow:https://stackoverflow.com/questions/77440091/databricks-unity-catalog-error-cannot-create-metastore-data-access/77506306#77506306

  • 0 kudos
JonLaRose
by New Contributor III
  • 2001 Views
  • 3 replies
  • 1 kudos

Delta Lake S3 multi-cluster writes - DynamoDB

Hi there!I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in...

  • 2001 Views
  • 3 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @JonLaRose , The multi-writers architecture for Delta Lake tables uses the S3 commit service.- The S3 commit service ensures write consistency across multiple clusters on a single table.- The service is part of the control plane and does not read ...

  • 1 kudos
2 More Replies
AClarkson
by New Contributor II
  • 4379 Views
  • 4 replies
  • 1 kudos

Resolved! AWS Secrets Manager access

HiWe're in the process of moving over from Databricks in Azure to AWS.I am trying to establish a method of accessing secrets from AWS Secrets Manager (we were using Azure KeyVault) and understand this can be done with boto as suggested from AWS.We ha...

  • 4379 Views
  • 4 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @AClarkson, I understand that you are trying to access AWS Secrets Manager from a shared cluster and are facing issues due to the constraints mentioned in the documentation . Since the instance metadata service (IMDS) is not accessible from the sh...

  • 1 kudos
3 More Replies
Masoomeh
by New Contributor II
  • 1436 Views
  • 1 replies
  • 0 kudos

Problem with provisioning lakehouse via Terraform

I am trying to provision the lakehouse on Azure using the Terraform template provided here: https://github.com/databricks/terraform-databricks-examples/tree/main/examples/adb-lakehouse. I have started from scratch and have only added a resource group...

  • 1436 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Masoomeh, I’m sorry to hear that you’re facing issues while provisioning the lakehouse on Azure using the Terraform template provided by Databricks. Based on the error messages you’ve shared, it seems that the metastore is not assigned to the cur...

  • 0 kudos
NateJ
by New Contributor II
  • 2141 Views
  • 5 replies
  • 2 kudos

Failed to start cluster: Large docker image

I have a large Docker image in our AWS ECR repo. The image is 27.4 GB locally and 11539.79 MB compressed in ECR.The error from the Event Log is:Failed to add 2 containers to the compute. Will attempt retry: true. Reason: Docker image pull failureJSON...

  • 2141 Views
  • 5 replies
  • 2 kudos
Latest Reply
amoghjain
New Contributor II
  • 2 kudos

I have a similar problem. a 10gb image pulls fine but a 31gb image doesnt. both workers and drivers have 64gb memory. i get the timeout error with "Cannot launch the cluster because pulling the docker image failed. Please double check connectivity fr...

  • 2 kudos
4 More Replies
Ishmael
by New Contributor III
  • 3250 Views
  • 7 replies
  • 3 kudos

Resolved! Connect to databricks from external non-spark cluster

Hi,I have an app/service on a non-spark kubernetes cluster. Is there a way to access/query a databricks service from my app/service? I see documentations on connectors, particularly on scala which is the code of my app/service. Can I use these connec...

  • 3250 Views
  • 7 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Ishmael, Yes, you are correct. ScalaPy is a library that enables seamless interoperability between Scala and Python. It is based on the Py4J project, which allows Python programs running in a Python interpreter to dynamically access Java objects ...

  • 3 kudos
6 More Replies
MasJamei
by New Contributor II
  • 957 Views
  • 2 replies
  • 0 kudos

Notebook Id level of uniqueness

Hi there,We know that notebook ids are unique. https://docs.databricks.com/en/workspace/workspace-details.html but I want to know in what level they're unique. For example, if Notebook Ids are unique within a workspace, or are they universally unique...

  • 957 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @MasJamei , This article explains how to get workspace, cluster, directory, model, notebook, and job identifiers and URLs in Databricks.

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels