- 58 Views
- 1 replies
- 1 kudos
Question about MySQL Integrated‑CDC pipeline on Classic compute workspace
Hi everyone,I am trying to set up an Integrated‑CDC pipeline for a MySQL RDS instance on our Databricks workspace. Our workspace only uses Classic compute; Serverless compute is not available.According to the documentation, Integrated‑CDC for MySQL s...
- 58 Views
- 1 replies
- 1 kudos
- 1 kudos
@david888 the missing wizard is expected. MySQL Integrated CDC currently runs on classic compute only, and UI-based pipeline creation isn't available. Workspace-level feature enablement is required; the docs direct you to your Databricks account team...
- 1 kudos
- 54 Views
- 1 replies
- 1 kudos
Which Topics Deserve More Attention Before the Databricks Data Engineer Associate Certification Exam
As soon as I began my preparation for the Databricks Data Engineer Associate Certification exam, I came to the realisation that merely going through the syllabus won’t be sufficient. There are certain topics which are recurrent in practical situation...
- 54 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @AdamLucas !Based on my personal experience, I can say that there are several practical areas that are more likely to be well-represented in the test. First, to understand and effectively operate Delta Lake, one has to have at least some general ...
- 1 kudos
- 259 Views
- 2 replies
- 1 kudos
Resolved! Storage Credential creation fails with "Access Connector ... could not be found"
Subject: Storage Credential creation fails with "Access Connector ... could not be found" despite fully verified Azure configurationHi all,I'm trying to create a storage credential (Azure Managed Identity) for Unity Catalog on a Premium Azure Databri...
- 259 Views
- 2 replies
- 1 kudos
- 1 kudos
@Data_Engineer14, Great troubleshooting so far. The fact that every Azure-side check passes cleanly but the storage credential creation still fails is a strong signal that this is a backend registration issue, not a misconfiguration on your end. Wh...
- 1 kudos
- 273 Views
- 4 replies
- 2 kudos
Incremental Load Issue with SDP
We have identified a critical issue in our pipeline and wanted to share it here to see if others have faced the same and how they approached it.Pipeline architecture: Bronze (Lakeflow Connect) → Silver (SCD2) → GoldThe Problem: Every pipeline run tri...
- 273 Views
- 4 replies
- 2 kudos
- 2 kudos
Thank you all for the responses. We could get into a solution by adding readChangeFeed flag as true as additional option to the spark.readStream however kept the pipeline configs same.
- 2 kudos
- 390 Views
- 5 replies
- 9 kudos
Resolved! Databricks Dashboard Pivot Table: default collapsed state?
I'm building a financial dashboard in Databricks using a Pivot visualization with two dimensions in the row hierarchy.The drill-down works as expected, but I'm having a usability issue: every time the view is loaded, both dimensions are fully expande...
- 390 Views
- 5 replies
- 9 kudos
- 9 kudos
Thanks everyone for the input.Marking @data_pulse's reply as the solution: it confirms through testing that the expand/collapse state isn't persisted in the published dashboard, so there's currently no setting for an initial collapsed state or a defa...
- 9 kudos
- 64 Views
- 1 replies
- 1 kudos
Auto-TTL on pipeline streaming table not firing
We have Auto-TTL configured on a Lakeflow pipeline streaming table via create_streaming_table(auto_ttl={"timestamp_column": "_discontinued_on", "expire_in_days": 77}). The autottl.timestampColumn and autottl.expireInDays properties are correctly set ...
- 64 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,You did the homework, so I'll skip repeating it and add three things.First, the docs contradiction you found is real, and it's worse than you listed. The GDPR page says "The recommended way is Predictive optimization for Unity Catalog managed tabl...
- 1 kudos
- 112 Views
- 3 replies
- 1 kudos
Resolved! ForEach Repair behaviour
What is the behaviour of a 'repair' run on a ForEach task?Will it only repair failed tasks & non-executed tasks (if cancelled early)? Or rerun all tasks? Where is this documented? Can this be added to the documentation of this task? https://docs.data...
- 112 Views
- 3 replies
- 1 kudos
- 1 kudos
Behavior: A repair run on a job containing a For Each task follows the same rule as repair runs generally — it re-executes only the unsuccessful task runs (failed, or skipped/not-executed due to early cancellation or an upstream failure) and any task...
- 1 kudos
- 879 Views
- 5 replies
- 7 kudos
Autoscaling with the autoloader without SDP
Hi there,I have a question regarding the autoloader without SDP and auto-scaling of clusters. I'm reading the following in the docs:Production considerations for Structured Streaming | Databricks on AWS:Do not enable autoscaling for compute for Struc...
- 879 Views
- 5 replies
- 7 kudos
- 7 kudos
Hi,Two clarifications that I think answer both of your questions.Why the docs say no to autoscaling on streaming jobs. Classic autoscaling only removes a node when it's idle (or, on Premium, "underutilized over the last 40 seconds" on jobs compute, j...
- 7 kudos
- 117 Views
- 4 replies
- 2 kudos
Resolved! Unity Catalog Volume as spark checkpoint location (in 2026)
Hi folks,I'd like my checkpoint folder to be easily accessible to me in my UC volume files. This was possible in Fabric OneLake (Files). For whatever reason, it is not easy in databricks volumes. I keep getting meaningless errors: spark.sparkConte...
- 117 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi,Glad it helped. On 14.3, a few things to keep straight before you pick plan B.The error you got isn't about managed vs external. sc.setCheckpointDir hands the path to the JVM's local file API (that java.io.File canonicalize in the stack), and the ...
- 2 kudos
- 60 Views
- 1 replies
- 1 kudos
Can the 5% deletion-vector threshold used by OPTIMIZE be configured?
Hello everyone, we have a compliance requirement to guarantee that deleted records are physically removed from the underlying Parquet files.Our current flow is:DELETE creates deletion vectors (DVs)OPTIMIZE may rewrite files and materialize the dele...
- 60 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,Short version: I couldn't find that knob anywhere in the Databricks docs. maxDeletedRowsRatio is an OSS Delta conf, and Databricks doesn't document it, so even if it's accepted on a given runtime you'd be relying on undocumented behavior for a com...
- 1 kudos
- 3949 Views
- 3 replies
- 2 kudos
Resolved! Serverless Compute - How to determine if being used programatically
Hi, We use a common notebook for all our "common" settings, this notebook is called in the first cell of each notebook we develop. This issue we are now having is that we need 2 common notebooks, one for a normal shared compute and one for serverle...
- 3949 Views
- 3 replies
- 2 kudos
- 2 kudos
Here's a better solution I found:import os on_serverless = 'IS_SERVERLESS' in os.environ and os.environ['IS_SERVERLESS'] == 'TRUE'
- 2 kudos
- 49 Views
- 0 replies
- 0 kudos
Unity Catalog v.2.0?
Are there plans for a v.2.0 of Unity Catalog? I find the organization of tables in databricks to be very constrained and inflexible. Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...
- 49 Views
- 0 replies
- 0 kudos
- 3179 Views
- 5 replies
- 0 kudos
Resolved! Download all pages of a multi-page dashboard
Hi,I have created a multi-page dashboard in databricks. I want to download all the pages of the dashboard as a single pdf file. But when i export the dashboard I get it only in .json format. Is there a way to download all the pages as a pdf file?
- 3179 Views
- 5 replies
- 0 kudos
- 0 kudos
In subscription setup there is limitation of PDF size should be less than 9 MB, So if size increase you'll not get all the pages in email.
- 0 kudos
- 446 Views
- 6 replies
- 1 kudos
Agent outside databricks communication with databricks delta table
Hello community,I have following use case in my project:User[ Ask any query in simple english related to data] -> AI Agent -> Databricks unity catalog -> Delta table.Currently required data for project is in volume of workspace. Then we apply medalli...
- 446 Views
- 6 replies
- 1 kudos
- 1 kudos
@Kushal_2612 The slowness you are seeing is generally when an external service submits queries against standard workspace compute or interactive clusters, which carry heavy execution overhead. For querying Unity Catalog Delta tables from an external ...
- 1 kudos
- 112 Views
- 1 replies
- 0 kudos
constraints are not creating on materialized views
Hi,We are loading data from the Bronze layer to the Silver layer and creating materialized views with primary key and foreign key constraints. We have a total of 33 dependent tables, and the load is being performed using a Spark Declarative Pipeline ...
- 112 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,"FAILED unexpectedly, please contact Databricks support" is the generic wrapper, the real cause is in the event logSELECT timestamp, level, message, error, details FROM event_log('your-pipeline-id') WHERE error IS NOT NULL OR level = 'ERROR' ORDER...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
