- 251 Views
- 1 replies
- 0 kudos
Advanced Learning Festival (15 June - 06 July 2026) - Voucher Eligibility Confirmation
Hello @Jim ,I completed all four modules of Learning Pathway 1 (Associate Data Engineering) for the Advanced Learning Festival on July 6, 2026, the published end date of the event. My Academy account email is [datasci.patel.rupesh@gmail.com]. Screens...
- 251 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rjcrystal786, Thanks for checking. Based on the published guidance for the Advanced Learning Festival, if all four modules in Learning Pathway 1 were completed within the event window, your 50% certification voucher will be sent to the email addr...
- 0 kudos
- 388 Views
- 4 replies
- 4 kudos
Databricks Standard SharePoint Connector Performance Issues
I've recently started using the Databricks Standard SharePoint connector within my workspace and have run into some significant performance issues.My notebook does a straightforward read using the following:lakeflow_connection_name = 'sharepoint_dev'...
- 388 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi all, thanks for the help.Unfortunately pathGlobFilter didn't improve performance for me.I ended up working around it with some pre-processing to figure out which specific folders I actually needed, then looping through those directories with the c...
- 4 kudos
- 193 Views
- 1 replies
- 1 kudos
Lakebase CDF — destination Delta table not created after successful UI setup (Free Edition)
Summary:We configured Lakebase CDF to stream changes from a native Postgres table (public.query_logs) into Unity Catalog, but the destination Delta table never appears despite completing all documented prerequisites.Steps taken1. Created query_logs ...
- 193 Views
- 1 replies
- 1 kudos
- 1 kudos
Most likely this is a Free Edition storage limitation rather than a bug, and your Postgres-side setup sounds correct. Lakebase CDF writes its destination as a Unity Catalog managed Delta table, and the Lakebase CDF docs list catalogs backed by defaul...
- 1 kudos
- 340 Views
- 2 replies
- 0 kudos
StatusCode.UNIMPLEMENTED error: DatabricksConnect library using AKS/PySpark to calling Spark cluster
I am running PySpark application in AKS/Pythgon container/pod:Using Databricks 18.2.1 library with Databricks Spark cluster 18.2Once a while I am getting below error: InactiveRpcError of RPC that terminated with: status = StatusCode.UNIMPLEMENTED det...
- 340 Views
- 2 replies
- 0 kudos
- 0 kudos
Short answer: this looks more like an intermittent Spark Connect transport/routing issue than a Spark job logic issue. Databricks Connect uses gRPC over HTTP/2, and the specific InactiveRpcError ... UNIMPLEMENTED ... Received http2 header with status...
- 0 kudos
- 134 Views
- 1 replies
- 0 kudos
Automation for any script changes in databricks and bit bucket
If any code changes are done and moved to bit bucket based on env now i want to fetch all the code changes and move all these changes to a table in adls/unity catalog table.Anyone please help with the approaches we are using bit bucket for repository
- 134 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, couple ways to do this depending on how tightly you want it wired in. If Bitbucket is just your source repo and you want a row per change, the simplest path is a Bitbucket Pipeline that runs on push. In that pipeline step, pull the commit metada...
- 0 kudos
- 1017 Views
- 14 replies
- 6 kudos
Resolved! PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON
I'm working on a personal data engineering project using Kafka, Spark Structured Streaming, and Docker.The application consumes two Kafka topics that originate from an external market-data websocket source:a trade streama candlestick (kline/OHLCV) st...
- 1017 Views
- 14 replies
- 6 kudos
- 6 kudos
Spark Structured Streaming writes to file sinks and generally it uses a phased commit by writing temporary files to the output directory followed by writing metadata with references and a final commit by moving/renaming temp files to final names. You...
- 6 kudos
- 411 Views
- 2 replies
- 3 kudos
Resolved! Spark UI Troubleshooting: Data Skew vs Cluster Resource Bottlenecks
How can Spark UI metrics be used to distinguish data skew from insufficient cluster resources?When a Databricks job is slow, we usually look at Spark UI metrics such as task duration, shuffle read/write, spilled bytes, GC time, executor CPU utilizati...
- 411 Views
- 2 replies
- 3 kudos
- 3 kudos
A simple way to troubleshoot a slow Spark job using Spark UI is:Check task durationA few very slow tasks → Likely data skew.Most tasks are slow → Likely cluster resource or execution issue.Check Spark UI metricsLarge differences in shuffle read/task ...
- 3 kudos
- 145 Views
- 1 replies
- 0 kudos
CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears
[CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND] Checkpoint block not found! Either the executor that originally checkpointed this partition is no longer alive, or the original RDD is unpersisted.After switching from reliable checkpoint() to localCheckpoint() to ...
- 145 Views
- 1 replies
- 0 kudos
- 0 kudos
Its a issue due to the fundamental difference in the handling of data durability by checkpoint and localCheckpoint. Provisioning a larger cluster will not reliably solve this problem as the issue is about executor lifecycle & not capacity.localCheckp...
- 0 kudos
- 464 Views
- 2 replies
- 3 kudos
Resolved! Detecting Photon fallback in-cluster + safe right-sizing from system tables
I'm prototyping a cluster cost / right-sizing advisor and wanted to get a reality-check from people running Databricks at real scale before I sink more time into it.The main thing I'm chasing is Photon fallback. Photon quietly drops to the JVM on uns...
- 464 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @Yogasathyandrun , I did some digging and would like to share some thoughts that you hopefully find useful. You've mapped the boundary here more accurately than most people do, so let me give you a quick reality check on your four sticking points...
- 3 kudos
- 281 Views
- 4 replies
- 0 kudos
SDP continuous mode
Hi,I was doing a POC and hence used open source spark and kafka in docket container and got it working. The sample code is ingesting data from kafka but it is running only in batch mode. Not able to continuously ingest the kafka streamQuestion: Can w...
- 281 Views
- 4 replies
- 0 kudos
- 0 kudos
Sample code that i usedfrom pyspark import pipelines as dp from pyspark.sql import DataFrame, SparkSession, functions as f from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DecimalType spark = SparkSession.active() @dp...
- 0 kudos
- 258 Views
- 2 replies
- 2 kudos
Lakeflow SDP (DLT) produce external tables, or only UC-managed
As I understand it, streaming tables and materialized views produced by Lakeflow Spark Declarative Pipelines (DLT) are always Unity Catalog managed tables , there's no LOCATION/path option on create_streaming_table or apply_changes.Is that correct? A...
- 258 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @nidhin, What you’re saying is basically correct for a Unity Catalog-enabled Lakeflow Spark Declarative Pipelines setup. In that model, pipelines publish streaming tables and materialized views into the target catalog and schema, the data is store...
- 2 kudos
- 778 Views
- 4 replies
- 5 kudos
Resolved! json file existing in volume but not showing in UI
I have some json files existing in a specific volume when I try to search for them they don't appear but when I query the the volume using python I am able to get them and read their content.Any help ?
- 778 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi,The global workspace search won't return results for files stored in Unity Catalog Volumes. Its indexing is focused on workspace assets and catalog-managed objects, rather than the underlying files within a Volume.To locate files in a Volume, navi...
- 5 kudos
- 851 Views
- 5 replies
- 2 kudos
Resolved! Databricks unable to list ADLS folder and files
Hi Databricks Community,I am able to list the container from my databricks workspace but unable to list the folder and files further.If I try to access the same files and folder from the Databricks UI, external location path, I am able to see all fil...
- 851 Views
- 5 replies
- 2 kudos
- 2 kudos
Following are may be the Causes1. Different authentication methods- The UI's external location uses Unity Catalog credentials- Your dbutils.fs.ls() command uses the compute's Spark configurations- These may be using different credentials with differe...
- 2 kudos
- 488 Views
- 1 replies
- 2 kudos
Resolved! DESCRIBE HISTORY Performance Issue for Large Scale Tables (22K Tables)
Hi everyone, I’m working with around 22,000 Unity Catalog external Delta tables, and my requirement is to execute DESCRIBE HISTORY table_name LIMIT 1 for each table and append the latest record into a single consolidated table. I’ve already tried mul...
- 488 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi,The reason your performance degrades so badly (4 mins for 2k tables, but 50 mins for 12k) is because of the Spark Driver. When you run spark.sql("DESCRIBE HISTORY...") inside a ThreadPoolExecutor, every single one of those 22,000 queries has to be...
- 2 kudos
- 293 Views
- 1 replies
- 1 kudos
databricks-connect library for python and pandas 3
Hello,databricks-connect is pinning pandas during the installation. Since we're moving towards pandas 3 can you please add the support for the newest version as well?
- 293 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Albertino , how are you doing today?Thanks for calling this out. as per my understanding, at the moment, the pandas pin is there intentionally. Databricks Connect release notes say supported pandas versions are currently limited to 1.0.5<=pandas<...
- 1 kudos