- 1168 Views
- 1 replies
- 0 kudos
Assistance Needed with Databricks DQX Framework for Data Quality
Hi Community Experts,I hope this message finds you well. Our team is currently working on enhancing data quality within our Databricks environment and we are utilizing the Databricks DQX framework for this purpose. We are seeking detailed guidance a...
- 1168 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @shubham_007! It looks like this post duplicates the one you recently posted. A response has already been provided there. I recommend continuing the discussion in that thread to keep the conversation focused and organised
- 0 kudos
- 1625 Views
- 3 replies
- 0 kudos
The case of the phantom files!
I developed a notebook that uses *.py files as module imports. On my cluster, the logic in the notebook works fine, my custom modules get loaded, code executes.Using an AzDO pipeline, I deploy the notebook and supporting files to a separate workspac...
- 1625 Views
- 3 replies
- 0 kudos
- 0 kudos
Got it.My background is not SWE, I have always been a 'data guy', but I definitely appreciate a proper dev workflow (ci/cd, git integration, tests).When we started using databricks like 7 or 8 years ago, we went for notebooks as this got us up to spe...
- 0 kudos
- 3602 Views
- 3 replies
- 2 kudos
UI menu customisation
If I want to customize the UI menu so some users/groups can't for example create jobs or make experiments etc.As I see it, when unity catalog is enabled, everyone can create jobs (If they have attach to cluster permission). But in my organization, th...
- 3602 Views
- 3 replies
- 2 kudos
- 2 kudos
I would also like a way to customize what users see when they log in. For example we don't want most of them to even have the option of creating a Genie Space so ideally we could simply remove "Genie" from their menu
- 2 kudos
- 6175 Views
- 11 replies
- 14 kudos
Smaller dataset causing OOM on large cluster
I have a pyspark job reading the input data volume of just ~50-55GB Parquet data from a delta table on Databricks. Job is using n2-highmem-4 GCP VM and 1-15 worker with autoscaling on databricks. Each workerVM of type n2-highmem-4 has 32GB memory and...
- 6175 Views
- 11 replies
- 14 kudos
- 14 kudos
Next, use the repartition(n) to increase your dataframe to twice the number you got earlier. For example, if num_partitions was 30, then repartition(60) prior to running your query. With half the data in each Memory Partition, I'm guessing you won't...
- 14 kudos
- 13101 Views
- 9 replies
- 1 kudos
How to move a metastore to a new Storage Account in unity catalog?
Hello, I would like to change the Metastore location in Databricks Account Console. I have one metastore created that is in an undesired container/storage account. I could see that it's not possible to edit a metastore that is already created. I coul...
- 13101 Views
- 9 replies
- 1 kudos
- 1 kudos
@TugrulA we repopulated our SQL warehouse by rerunning all Pipelines (Delta Live Tables) that read our raw data from an Azure storage account (different than the Metastore storage container) and write the bronze/silver/gold tables to the (new) Metast...
- 1 kudos
- 3581 Views
- 2 replies
- 0 kudos
Databricks Unity Catalog Metastore
Hey everyone,I deleted my Unity Catalog metastore and now want to point it to another Azure storage account (ADLS). However, once a metastore is created, its storage location cannot be changed. Therefore, I deleted the existing metastore and created ...
- 3581 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey TugrulA - 1. Deleting a Unity Catalog Metastore permanently removes all associated objects, and the new metastore wont automatically include original objects. Unfortunately automatic recover is not possible. While UC allows UNDROP for individual ...
- 0 kudos
- 1742 Views
- 2 replies
- 0 kudos
Resolved! Trying to understand why a cluster reports as "terminating" right after being created
We use a "warmup" mechanism to get our DBR instance pool into a state where it has at-least-N instances. The logic is:For N repetitions:Request a new DBR cluster in the pool (which causes the pool to request an AWS instance)Wait for the cluster to re...
- 1742 Views
- 2 replies
- 0 kudos
- 0 kudos
Aha, found it. I monitored the pool status via the DBR UI, and when a cluster *started* being provisioned, I clicked into it. Then I looked at the event log, and found useful information about failed steps. The underlying error was indeed AWS related...
- 0 kudos
- 3356 Views
- 3 replies
- 1 kudos
Resolved! Databricks AWS permission question
Hello,I'm currently using Databricks on AWS for some basic ETL where the resulting data is stored as Hive external delta tables.Even though Unity catalog is disabled, table access control is disabled for the workspace, and the cluster is running with...
- 3356 Views
- 3 replies
- 1 kudos
- 1 kudos
@Lennart Glad to hear it helped! If you think this solves your question, please consider marking it as the accepted answer so it can assist other users as well.Best regards, Isi
- 1 kudos
- 2128 Views
- 1 replies
- 0 kudos
Linking Workspace IDs to Names in Billing Schema
Hi everyone,We recently enabled UC and the Billing system table to monitor our usage and costs. We've successfully set up a dashboard to track these metrics for each workspace. The usage table includes the workspace_id, but I'm having trouble finding...
- 2128 Views
- 1 replies
- 0 kudos
- 0 kudos
I got this from their older version of the dashboard. dbdemos uc-04-system-tablesWhen everything is executed, go to your graph in the dashboard, click three dots in the top right, select in my case "View dataset:usage_overview" then paste/modify sql ...
- 0 kudos
- 1101 Views
- 1 replies
- 0 kudos
Unable to Access Data Apps/Templates in Databricks Workspace
Hi,We are currently unable to view any of the standard apps/templates (such as Data Apps, Chatbot, and Hello World) within our Databricks workspace.Workspace Details:Workspace Name: KdataaiWorkspace URL: https://1027317917071147.7.gcp.databricks.comC...
- 1101 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @aishwaryakdataa! Could you please verify if the Databricks Apps service is enabled in your workspace? You may also need to review the App configuration settings and ensure that the appropriate roles and permissions are in place to access the t...
- 0 kudos
- 2753 Views
- 3 replies
- 0 kudos
Unable to add a microsoft security group as Workspace Admin
I'm a workspace admin for a databricks workspace. I can add a microsoft security group in the workspace. When I click on the group to view it I can view the members of the group same in the Azure AD reflecting correctly but it throws an error on the ...
- 2753 Views
- 3 replies
- 0 kudos
- 0 kudos
@pranav5 This issue usually occurs because of how Databricks handles group provisioning via SCIM, especially with external groups from Azure AD.SCIM 404 Error: This generally means Databricks cannot find a matching SCIM identity for the Azure AD grou...
- 0 kudos
- 1907 Views
- 2 replies
- 0 kudos
JDBC Driver cannot connect when using TokenCachePassPhrase property
Hello all, I'm looking for suggestions on enabling the token cache when using browser based SSO login. I'm following the instructions found here: Databricks-JDBC-Driver-Install-and-Configuration-Guide For my users, I would like to enable the token ca...
- 1907 Views
- 2 replies
- 0 kudos
- 0 kudos
For the error encountered (Cannot invoke "java.nio.file.attribute.AclFileAttributeView.setAcl(...)" because "<local6>" is null) might be permission or file system issues where the token cache store is being accessed. When EnableTokenCache=0, the to...
- 0 kudos
- 5670 Views
- 3 replies
- 1 kudos
Databricks Notebook says "Connecting.." for some users
For some users, after clicking on a notebook the screen says "connecting..." and the notebook does not open.The users are using Chrome browser and the same happens with Edge as well.What could be the reason?
- 5670 Views
- 3 replies
- 1 kudos
- 1 kudos
Even I am facing the same issue. It always keep saying, opening the notepad. Luckily once it is opened and when connected with the cluster, then its getting timeout.
- 1 kudos
- 5011 Views
- 8 replies
- 1 kudos
Notebook Detached Error: exception when creating execution context: java.net.SocketTimeoutException:
Hello Community,I have been facing this issue since yesterday. After attaching the cluster to a notebook and running a cell, I get the following error in the community edition of the databricks:Notebook detached:exception when creating execution cont...
- 5011 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi All,I am also facing issue.if anyone knows how to resolve this, please post the solution here.
- 1 kudos
- 4185 Views
- 7 replies
- 0 kudos
preloaded_docker_images: how do they work?
At my org, when we start a databricks cluster, it oftens takes awhile to become available (due to (1) instance provisioning, (2) library loading, and (3) init script execution). I'm exploring whether an instance pool could be a viable strategy for im...
- 4185 Views
- 7 replies
- 0 kudos
- 0 kudos
Hello, when we specify docker image with credentials in instance pool configuration, should we also specify credentials in cluster configuration?. as we already have image pulled into the pool instance.
- 0 kudos
