- 1835 Views
- 4 replies
- 2 kudos
Resolved! count function
Hi, as per spark internals, once count function executed in worker nodes , one of the worker node collect all the count of records and do summation ?or count of records from all worker nodes passed to driver node. and summation done driver node side....
- 1835 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @neerajaN, You are correct that the count() operation follows a two-phase aggregation pattern in Spark. Here is how it works in detail: PHASE 1: PARTIAL AGGREGATION (EXECUTORS) Each executor computes a local partial count for the partitions assign...
- 2 kudos
- 1779 Views
- 3 replies
- 0 kudos
Resolved! Python segmentation fault in serverless job
We're getting a Python segmentation fault in a serverless job that uses Delta Table merge inside a foreachBatch step in structured streaming (trigger once)./databricks/python/lib/python3.12/site-packages/pyspark/sql/connect/streaming/query.py:479: Us...
- 1779 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Malthe, Since you have confirmed this is vanilla PySpark with no external libraries on serverless runtime environment version 5, this narrows things down considerably. Here are some additional observations and recommendations beyond what Louis sh...
- 0 kudos
- 2222 Views
- 3 replies
- 1 kudos
Resolved! Unable to access files using a classic cluster
I used the same code with the classic cluster (RunTime 17.3LTS ML, with spark config: "spark.databricks.workspace.fileSystem.enabled true"), not able to access files in workspace with the following python code: import os# Check if source exists and w...
- 2222 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @NW1000, This behavior comes down to how workspace file access and identity work differently between serverless compute and classic clusters. SERVERLESS COMPUTE Serverless interactive compute runs under your own identity. It inherits your workspac...
- 1 kudos
- 3118 Views
- 3 replies
- 3 kudos
Resolved! Conditional Logic in Databricks Asset Bundles using Go Templates
I am defining a job using Databricks Asset Bundles (DABs) as follows:YAML resources: jobs: job_name: ... schedule: {{ if eq ${var.env} "prd" }} pause_status: "UNPAUSED" {{ else }} pause_status: "PAUSE...
- 3118 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Seunghyun, Go template syntax ({{if}}, {{eq}}, etc.) is only supported in bundle project templates, which are the .tmpl files used during "databricks bundle init" to scaffold new projects. It is not supported inside your regular databricks.yml co...
- 3 kudos
- 2774 Views
- 2 replies
- 3 kudos
Resolved! Managing dashboard refresh schedules in DABs
I am currently using Databricks Asset Bundles (DABs) to deploy and manage dashboard resources. While I can manually add a schedule to a dashboard via the Databricks console, I would like to reflect this same configuration in the dashboard YAML file. ...
- 2774 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Seunghyun, You are correct that the dashboard resource definition in Databricks Asset Bundles does not currently include schedule-related properties. The dashboard resource supports properties like display_name, file_path, warehouse_id, embed_cre...
- 3 kudos
- 3055 Views
- 3 replies
- 2 kudos
Optimizing Large Materialized View to expedite query execution
Hi All, I have a DLT Pipeline setup which reading Parquets from S3 Bucket and create a materialized view. Created view is quite big and contains Billion of records and contain around few TB of data. Predictive Optimization is already enabled. automat...
- 3055 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @FAHADURREHMAN, There are several layers to optimizing query performance on a multi-TB materialized view, and the other replies here cover the ingestion/refresh side well. Let me add some guidance on the query-side tuning and help you decide betwe...
- 2 kudos
- 1968 Views
- 2 replies
- 1 kudos
Resolved! Stream to static join - late arriving records
I have a stream to static join, but some of the rows in the static table arrive later than the linked rows in the stream.What is the default behaviour if a record in the stream hasn't joined a record in the static table? Is it lost forever?How is thi...
- 1968 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @yit337, This is an important topic to understand, so let me walk through the mechanics in detail. HOW STREAM-STATIC JOINS WORK In a stream-static join, each micro-batch of streaming data is joined against the static DataFrame. The key behavior de...
- 1 kudos
- 2511 Views
- 3 replies
- 1 kudos
Resolved! Fatal error: The Python kernel is unresponsive.
I'm having problems with running databricks with cv2.Everytime I try to import cv2 I get this error. If I comment it out error disappears.
- 2511 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Innuendo84, Glad you got this resolved. For anyone else who runs into this, here is some additional context on why this happens and how to avoid it. THE ROOT CAUSE The standard opencv-python package includes GUI components (highgui, GTK/Qt bindin...
- 1 kudos
- 1336 Views
- 3 replies
- 1 kudos
Resolved! DLT Auto Loader Reading from Parent S3 Folder not Sub Folders
Hi All, I am trying to read csv files from one Folder of S3 bucket. For this particular used case, I do not intent to read from sub folders. I am using below code however its reading all CSVs in sub folders as well. How can i avoid that? I used many ...
- 1336 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @FAHADURREHMAN, This is expected behavior with Auto Loader. By default, when you point it at a directory path like s3://bucket/folder/, it will recursively traverse all subdirectories and pick up matching files. The pathGlobFilter option only filt...
- 1 kudos
- 1552 Views
- 2 replies
- 1 kudos
Resolved! DEEP CLONE fails with [UNRESOLVED_ROUTINE] Cannot resolve routine isNotNull on DBR 16.4
Hi Databricks Community, I'm encountering an issue when attempting to DEEP CLONE a Delta table on DBR 16.4 that works fine on DBR 13.3.Error Message: [UNRESOLVED_ROUTINE] Cannot resolve routine `isNotNull` on search path [`system`.`builtin`, `system`...
- 1552 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @dplatform_user, This error occurs because of how NOT NULL constraints are internally represented in Delta table metadata. When a Delta table has NOT NULL columns, the Delta protocol stores these as CHECK constraints using expressions like isNotNu...
- 1 kudos
- 1724 Views
- 2 replies
- 0 kudos
Resolved! Databricks Spark UI showing -1 Executors
Hi Community,This might be a basic question, but I’m asking for educational purposes.I noticed that in one of my jobs, the Spark UI shows -1 executors. Initially, I thought this might indicate that executors are idle, but that doesn’t seem to explain...
- 1724 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Kirankumarbs, The -1 value you are seeing for executors in the Spark UI depends on which type of compute your job is running on, so let me cover both scenarios. SERVERLESS COMPUTE If your job is running on serverless compute, this is the expected...
- 0 kudos
- 1766 Views
- 2 replies
- 2 kudos
Resolved! check statistics of clustering columns per file to see how liquid clustering works
I have a Delta table on which I set up liquid clustering using three columns. I would like to check file statistics to see how the clustering column values are distributed along the files. How can I write a query that shows min and max values, etc. o...
- 1766 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @smoortema, There are several approaches for inspecting per-file column statistics on a liquid-clustered Delta table. Here is a walkthrough from simplest to most detailed. APPROACH 1: CONFIRM CLUSTERING CONFIGURATION First, verify that clustering ...
- 2 kudos
- 2578 Views
- 3 replies
- 2 kudos
Resolved! Reading data from Serverless Warehouse from Azure Functions in Python - using managed identities
We are trying to run a simple service on an Azure Function app, where we need to query some data from a Databricks Warehouse. We want to avoid managing secrets, and hence try to use Microsoft Entra authentication all the way. Using various available ...
- 2578 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @MRTN, The 403 Forbidden error you are seeing when using DefaultAzureCredential from the Azure Function (while it works locally with AzureCliCredential) comes down to a key distinction in how the token is being used and who the token represents. U...
- 2 kudos
- 1132 Views
- 2 replies
- 0 kudos
Resolved! Freezing the Page Filter Section During Scroll in Databricks
I have a question regarding Databricks Dashboards.I would like to fix the filter area within the dashboard so that it remains visible at all times, even when scrolling up or down. Is there a way to make the Page Filter area 'sticky' or frozen, exclud...
- 1132 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Seunghyun, AI/BI Dashboards do not currently have a built-in option to make page-level filters sticky or frozen during scroll. However, there are a couple of approaches you can use to keep filters accessible for your viewers. OPTION 1: USE GLOBAL...
- 0 kudos
- 2719 Views
- 6 replies
- 6 kudos
Error reading an external table - when using a serverless compute.
I am trying to read an external table in Databricks, created and maintained by using the delta-rs Python module. This usually works just fine, but after a recent checkpoint generation, I get the error below. However, the error only appers when readin...
- 2719 Views
- 6 replies
- 6 kudos
- 6 kudos
Hi @MRTN, Errors reading external tables from serverless compute are almost always caused by how serverless handles cloud storage access compared to classic compute. Serverless compute runs in a Databricks-managed compute plane, so it cannot use DBFS...
- 6 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
