- 50 Views
- 1 replies
- 1 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 50 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @adhi_databricks !This is a very specific and subtle architectural edge case when migrating traditional Scala JAR workloads to Databricks Serverless Compute.The core issue comes down to how Databricks Serverless Jobs execute code via Spark Connect...
- 1 kudos
- 4090 Views
- 10 replies
- 3 kudos
Resolved! Unable to create secret scope -"Fetch request failed due expired user session"
Hi everyone,I’m trying to create an Azure Key Vault-backed secret scope in a Databricks Premium workspace, but I keep getting this error: Fetch request failed due expired user sessionSetup details:Databricks workspace: PremiumAzure Key Vault: Owner p...
- 4090 Views
- 10 replies
- 3 kudos
- 3 kudos
Hi @lofidev I resolved by requesting a policy exemption window to create scopes. The issue was due to a restrictive policy. Precisely what @lingareddy_Alva summarized earlier on this thread "When Databricks tries to programmatically grant its SP Get/...
- 3 kudos
- 339 Views
- 4 replies
- 1 kudos
Disable spark transformations inside try except warning
I'm wondering if there's a way to suppress certain errors? I have a lot of spammy yellow underlining from this one warning, and it doesn't seem that the warning behaves properly. Transformation inside try/except block is lazy, fine, I have actions on...
- 339 Views
- 4 replies
- 1 kudos
- 1 kudos
If your action (count(), collect(), show(), etc.) is already inside the try block, this sounds like a false positive. The warning is likely from a Databricks inspection rather than the Flint time-series library. Check if inspection SCPAP005 can be di...
- 1 kudos
- 18 Views
- 0 replies
- 0 kudos
Power BI Refresh Fails with "Unable to Continue Fetch After Reconnect" from Azure Databricks
Hi everyone,I'm refreshing a report with dataset (~10 million rows) from Azure Databricks in power Bi service. The refresh runs for about an hour and then fails with: Unable to continue fetch after reconnect. Retry limit exceeded. I am currently unab...
- 18 Views
- 0 replies
- 0 kudos
- 454 Views
- 6 replies
- 0 kudos
Resolved! VOID column inside STRUCT fails to cast to VARIANT
Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...
- 454 Views
- 6 replies
- 0 kudos
- 0 kudos
Your screenshot already uses TO_VARIANT_OBJECT; the VOID field inside the STRUCT is what causes the failure. TO_VARIANT_OBJECT requires every field to be convertible to VARIANT, but VOID is not accepted in that nested path. Scalar CAST(PAYMENT_DATE A...
- 0 kudos
- 68 Views
- 0 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 68 Views
- 0 replies
- 0 kudos
- 622 Views
- 3 replies
- 0 kudos
Advanced Learning Festival (15 June - 06 July 2026) - Voucher Eligibility Confirmation
Hello @Jim ,I completed all four modules of Learning Pathway 1 (Associate Data Engineering) for the Advanced Learning Festival on July 6, 2026, the published end date of the event. My Academy account email is [datasci.patel.rupesh@gmail.com]. Screens...
- 622 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ashwin_DSA ,Thank you for the confirmation. Following up as it's now July 17 and I haven't yet received the voucher at the email address associated with my Academy account (I've checked spam/promotions folders as well).For reference, I completed ...
- 0 kudos
- 522 Views
- 4 replies
- 4 kudos
Databricks Standard SharePoint Connector Performance Issues
I've recently started using the Databricks Standard SharePoint connector within my workspace and have run into some significant performance issues.My notebook does a straightforward read using the following:lakeflow_connection_name = 'sharepoint_dev'...
- 522 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi all, thanks for the help.Unfortunately pathGlobFilter didn't improve performance for me.I ended up working around it with some pre-processing to figure out which specific folders I actually needed, then looping through those directories with the c...
- 4 kudos
- 485 Views
- 1 replies
- 1 kudos
Resolved! Lakebase CDF — destination Delta table not created after successful UI setup (Free Edition)
Summary:We configured Lakebase CDF to stream changes from a native Postgres table (public.query_logs) into Unity Catalog, but the destination Delta table never appears despite completing all documented prerequisites.Steps taken1. Created query_logs ...
- 485 Views
- 1 replies
- 1 kudos
- 1 kudos
Most likely this is a Free Edition storage limitation rather than a bug, and your Postgres-side setup sounds correct. Lakebase CDF writes its destination as a Unity Catalog managed Delta table, and the Lakebase CDF docs list catalogs backed by defaul...
- 1 kudos
- 650 Views
- 2 replies
- 2 kudos
Resolved! StatusCode.UNIMPLEMENTED error: DatabricksConnect library using AKS/PySpark to calling Spark cluster
I am running PySpark application in AKS/Pythgon container/pod:Using Databricks 18.2.1 library with Databricks Spark cluster 18.2Once a while I am getting below error: InactiveRpcError of RPC that terminated with: status = StatusCode.UNIMPLEMENTED det...
- 650 Views
- 2 replies
- 2 kudos
- 2 kudos
Short answer: this looks more like an intermittent Spark Connect transport/routing issue than a Spark job logic issue. Databricks Connect uses gRPC over HTTP/2, and the specific InactiveRpcError ... UNIMPLEMENTED ... Received http2 header with status...
- 2 kudos
- 198 Views
- 1 replies
- 0 kudos
Automation for any script changes in databricks and bit bucket
If any code changes are done and moved to bit bucket based on env now i want to fetch all the code changes and move all these changes to a table in adls/unity catalog table.Anyone please help with the approaches we are using bit bucket for repository
- 198 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, couple ways to do this depending on how tightly you want it wired in. If Bitbucket is just your source repo and you want a row per change, the simplest path is a Bitbucket Pipeline that runs on push. In that pipeline step, pull the commit metada...
- 0 kudos
- 1462 Views
- 14 replies
- 6 kudos
Resolved! PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON
I'm working on a personal data engineering project using Kafka, Spark Structured Streaming, and Docker.The application consumes two Kafka topics that originate from an external market-data websocket source:a trade streama candlestick (kline/OHLCV) st...
- 1462 Views
- 14 replies
- 6 kudos
- 6 kudos
Spark Structured Streaming writes to file sinks and generally it uses a phased commit by writing temporary files to the output directory followed by writing metadata with references and a final commit by moving/renaming temp files to final names. You...
- 6 kudos
- 647 Views
- 2 replies
- 3 kudos
Resolved! Spark UI Troubleshooting: Data Skew vs Cluster Resource Bottlenecks
How can Spark UI metrics be used to distinguish data skew from insufficient cluster resources?When a Databricks job is slow, we usually look at Spark UI metrics such as task duration, shuffle read/write, spilled bytes, GC time, executor CPU utilizati...
- 647 Views
- 2 replies
- 3 kudos
- 3 kudos
A simple way to troubleshoot a slow Spark job using Spark UI is:Check task durationA few very slow tasks → Likely data skew.Most tasks are slow → Likely cluster resource or execution issue.Check Spark UI metricsLarge differences in shuffle read/task ...
- 3 kudos
- 251 Views
- 1 replies
- 0 kudos
CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears
[CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND] Checkpoint block not found! Either the executor that originally checkpointed this partition is no longer alive, or the original RDD is unpersisted.After switching from reliable checkpoint() to localCheckpoint() to ...
- 251 Views
- 1 replies
- 0 kudos
- 0 kudos
Its a issue due to the fundamental difference in the handling of data durability by checkpoint and localCheckpoint. Provisioning a larger cluster will not reliably solve this problem as the issue is about executor lifecycle & not capacity.localCheckp...
- 0 kudos
- 589 Views
- 2 replies
- 3 kudos
Resolved! Detecting Photon fallback in-cluster + safe right-sizing from system tables
I'm prototyping a cluster cost / right-sizing advisor and wanted to get a reality-check from people running Databricks at real scale before I sink more time into it.The main thing I'm chasing is Photon fallback. Photon quietly drops to the JVM on uns...
- 589 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @Yogasathyandrun , I did some digging and would like to share some thoughts that you hopefully find useful. You've mapped the boundary here more accurately than most people do, so let me give you a quick reality check on your four sticking points...
- 3 kudos