cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Vivek_Singh
by • New Contributor III
  • 315 Views
  • 1 replies
  • 0 kudos

Getting error :USER_DEFINED_FUNCTIONS.CORRELATED_REFERENCES_IN_SQL_UDF_CALLS_IN_DML_COMMANDS_NOT_IMP

Hello Focus,need help, implemented Row level security at Unity Catalog, it is working as expected however while deleting the record getting error as enclosed detail [USER_DEFINED_FUNCTIONS.CORRELATED_REFERENCES_IN_SQL_UDF_CALLS_IN_DML_COMMANDS_NOT_IM...

Data Engineering
Reply
  • 315 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

The correlated subqueries within SQL User-Defined Functions (UDFs) used for row-level security are currently not supported for DELETE operations in Unity Catalog. You will need to adjust your row_filter_countryid_source_table UDF to avoid correlated ...

  • 0 kudos
SankaraiahNaray
by • New Contributor II
  • 2653 Views
  • 1 replies
  • 1 kudos

default auth: cannot configure default credentials

 I'm trying to use dbutils from WorkspaceClient and i tried to run this code from databricks notebook.But i get this errorError:ValueError: default auth: cannot configure default credentials Code:from databricks.sdk import WorkspaceClientw = Workspac...

Data Engineering
Reply
  • 2653 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 1 kudos

To resolve the ValueError: default auth: cannot configure default credentials error when using dbutils from WorkspaceClient in a Databricks notebook, follow these steps: Ensure SDK Installation: Make sure the Databricks SDK for Python is installed. ...

  • 1 kudos
SakuraDev1
by • New Contributor II
  • 417 Views
  • 1 replies
  • 0 kudos

autoloader cache and buffer utilization error

Hey guys,I'm encountering an issue with a project that uses Auto Loader for data ingestion. The production cluster is shutting down due to the error: The Driver restarted - possibly due to an OutOfMemoryError - and this stream has been stopped.I’ve i...

SakuraDev1_0-1729271704783.png SakuraDev1_0-1729271834424.png
Data Engineering
Reply
  • 417 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

The error message is sometimes generic "possibly due to an OutOfMemoryError" There is memory pressure indeed, but try to correlate those graph metrics with the Driver's STDOUT file content and check if the GC/FullGCs are able to work properly and rec...

  • 0 kudos
SakuraDev1
by • New Contributor II
  • 354 Views
  • 1 replies
  • 0 kudos

SakuraDev1 / Board: data-engineering (39000)

Link to post: (autoloader cache and buffer utilization error)by SakuraDev1 https://community.databricks.com/t5/data-engineering/autoloader-cache-and-buffer-utilization-error/m-p/94927#M39000 Hey guys, I'm encountering an issue with a project that use...

Data Engineering
Reply
  • 354 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

To address the resource scheduling and code-specific optimizations for your Auto Loader data ingestion pipeline, consider the following suggestions: Resource Scheduling Dynamic Allocation: Enable dynamic allocation in your cluster configuration. Thi...

  • 0 kudos
pesky_chris
by • New Contributor III
  • 529 Views
  • 1 replies
  • 0 kudos

Resolved! Support of Dashboards in Databricks Asset Bundles

Hello Databricks & Fellow Users,I noticed that support for Dashboards in DABs is coming soon (per the recent Databricks CLI pull request). Does anyone know if there are additional features planned to enhance the dashboard lifecycle? Currently, Git Fo...

Data Engineering
Reply
  • 529 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

I can see that the usage of Git/Repos in Lakeview Dashboards is already in development, there is no ETA yet of when this will be GA but we can guarantee it is in progress  

  • 0 kudos
cool_cool_cool
by • New Contributor II
  • 774 Views
  • 1 replies
  • 0 kudos

Databricks Workflow is stuck on the first task and doesnt do anyworkload

Heya I have a workflow in databricks with 2 tasks. They are configured to run on the same job cluster, and the second task depends on the first.I have a weird behavior that happened twice now - the job takes a long time (it usually finishes within 30...

Data Engineering
Reply
  • 774 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Given the provided context, the suggestion is to capture thread dumps from both the Spark Driver and any Active Executor when the task seems to be hung. Ideally, you should also be able to find in the Spark logs for the active executor with the hung ...

  • 0 kudos
Dave_Nithio
by • Contributor II
  • 675 Views
  • 1 replies
  • 0 kudos

Production vs Development DLT Schema

My organization is currently ingesting data utilizing a Delta Live Table pipeline. This pipeline points to a production Storage location and Target schema. This means that whenever we make changes to this pipeline, it directly impacts the production ...

Data Engineering
Reply
  • 675 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

To test changes to your Delta Live Table (DLT) pipeline without impacting production data, you can point to a different storage location and target schema. This does not require creating a completely separate DLT pipeline. Here are the steps: Create...

  • 0 kudos
adhi_databricks
by • New Contributor III
  • 309 Views
  • 1 replies
  • 0 kudos

DATABRICKS CLEANROOMS

Hi Team,I have a few questions regarding Databricks Cleanrooms:For onboarding first-party data, does the collaborator need a Databricks account with an enabled UC workspace?How is it useful for activating data for retargeting or prospecting use cases...

Data Engineering
Reply
  • 309 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

For onboarding first-party data, the collaborator does need a Databricks account with an enabled Unity Catalog (UC) workspace. This is necessary to map system tables into its metastore and to observe non-UC governed assets. Activating data for retarg...

  • 0 kudos
sanket-kelkar
by • New Contributor II
  • 496 Views
  • 1 replies
  • 0 kudos

Auto OPTIMIZE causing a data discrepancy

I have a delta table in Azure Databricks that gets MERGEd every 10 minutes.In the attached screenshot, in the version history of this table, I see a MERGE operation every 10 minutes which is expected. Along with that, I see the OPTIMIZE operation aft...

Data Engineering
Reply
  • 496 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Can you please provide more context about this, specifically with respect to the DBR Release and reproducibility of this scenario? Any metrics or plan change differences between both select statements, while the Optimize was in progress and after? Th...

  • 0 kudos
AcrobaticMonkey
by • New Contributor II
  • 377 Views
  • 1 replies
  • 0 kudos

Cannot Get Query Results in SQL Alerts

Example Query:select name, date from errors;Now i want to trigger an alert if count is greater than 1, and a notification should be sent to slack with output rows (name and date values). Even if i use {{QUERY_RESULT_ROWS}}, it only gives value after ...

Data Engineering
Reply
  • 377 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

Note I have not tried this myself, but can you try the following and let me know if it helps: Create the query, so SELECT name, date FROM errors;Set up the alert. Set the condition to trigger the alert when the count of rows is greater than 1.Create ...

  • 0 kudos
jonathanjone
by • New Contributor
  • 332 Views
  • 1 replies
  • 0 kudos

Facing Some Issues with Tablet PC and Databricks Product – Any Advice?

Hello everyone,I’m having some trouble using Databricks SQL Analytics v2.1 on my tablet PC, and I was wondering if anyone here has had similar experiences or could offer some advice.The main issues I’m facing are:Performance Slowdowns: When I run com...

Data Engineering
Reply
  • 332 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
NandiniN
Databricks Employee
  • 0 kudos

Hi @jonathanjone , 1 - Performance Slowdowns could be because of the warehouse size and the query count, if there are many queries, the warehouse has a limit of 10 query in parallel, else you see the query being queued.  You could also check if the q...

  • 0 kudos
guangyi
by • Contributor III
  • 378 Views
  • 1 replies
  • 1 kudos

Resolved! Has the numUpdateRetryAttempts property been deprecated?

I noticed there is a numUpdateRetryAttempts property mentioned in the the document https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/properties  used for configing the retry times of any DLT pipeline, but I cannot find it in the DL...

Data Engineering
Reply
  • 378 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 1 kudos

According to the Delta Live Tables properties reference, pipelines.numUpdateRetryAttempts is a recognized configuration parameter. It specifies the maximum number of attempts to retry an update before failing the update when a retryable failure occur...

  • 1 kudos
Viswanth
by • New Contributor II
  • 996 Views
  • 3 replies
  • 0 kudos

Implementing Conditional Logic for Dependent Tasks Using SQL Output and Task Values

Hi team,I'm working on setting up a workflow with task dependencies where a subsequent task should execute conditionally, based on the result of a preceding SQL task. Specifically, I need to evaluate an if/else condition on the output of the SQL quer...

Data Engineering
Reply
  • 996 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Ramana
Contributor
  • 0 kudos

This feature is in the Private Preview.

  • 0 kudos
2 More Replies
dbr_data_engg
by • New Contributor II
  • 716 Views
  • 1 replies
  • 1 kudos

Unable to deploy Databricks Asset Bundle

Hi Team,I created a workflow/job and was able to deploy to Dev and Prod workspaces. But now I am unable to deploy job to "Dev" workspace and getting below error, [Also unable to see this job on Databricks UI]Deploying resources...Updating deployment ...

Data Engineering
Reply
  • 716 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
fabiobeider
New Contributor II
  • 1 kudos

Hey, I'm facing the same issueDid you ever get a chance to solve it?

  • 1 kudos
emiliec
by • New Contributor
  • 427 Views
  • 1 replies
  • 0 kudos

QGIS python command in Databricks notebook

Hello, I would like to run a qgis python script in a databricks notebook. Currently, databricks doesn't recognize the qgis package. For example, i'd like to run this small example : from qgis.core import *# Supply path to qgis install locationQgsAppl...

Data Engineering
Reply
  • 427 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
VZLA
Databricks Employee
  • 0 kudos

QGIS is not directly usable in Scala or Spark environments as it is a standalone Geographic Information System (GIS) application, installing and using it directly within Databricks may not be straightforward due to the specific environment and depend...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels