- 342 Views
- 11 replies
- 0 kudos
Change data feed from a materialized view
Hi everyone,Trying the CDF on materialized view (declarative pipeline) feature that is currently in Beta.- Configuration:** Verified the MVs (materialized veiws) have: 'delta.enableRowTracking'SHOW TBLPROPERTIES my_mv ('delta.enableRowTracking'); re...
- 342 Views
- 11 replies
- 0 kudos
- 0 kudos
The feature that will help the most is the CDF on MV that is in beta right now. I have been testing it to get ready at my side and take advantage of it once it is in General Availability. The Automatic change data feed on Streaming tables would be us...
- 0 kudos
- 86 Views
- 1 replies
- 2 kudos
Resolved! Streaming read fails with "Detected a data update" after a restatement job touches the source
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
- 86 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Islam_hoti ,You have hit the classic 'Streaming vs. Mutation' wall. readStream on a Delta table assumes an append-only sequence; once an UPDATE happens, the commit versioning sequence is invalidated, hence the loud failure.Here is how we reasoned...
- 2 kudos
- 126 Views
- 3 replies
- 5 kudos
Resolved! Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...
- 126 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Islam_hoti ,Photon is fantastic, but it’s an 'all-or-nothing' value proposition. Once you hit a fallback, you lose the vectorized execution benefit for that entire sub-tree, and you're still paying the DBU premium. Here is how we approach auditin...
- 5 kudos
- 124 Views
- 3 replies
- 4 kudos
Resolved! At what data size do you stop reaching for Spark?
Hi everyone,A question I keep having with my team and I would like to hear how others think about it.A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is w...
- 124 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Islam_hoti ,This is the 'Silent Architect' dilemma—when the standard tool is technically suboptimal but organizationally necessary. We’ve wrestled with this, and here’s where we landed:The 'Standardization Premium' is real: We reasoned that the c...
- 4 kudos
- 3725 Views
- 5 replies
- 9 kudos
Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition
As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks F...
- 3725 Views
- 5 replies
- 9 kudos
- 9 kudos
Das ist ein beeindruckendes Projekt, besonders die Kombination aus Stimmungsanalyse, Embeddings und personalisierten Filmempfehlungen. Xtream Codes bestehen aus Server Adresse, Benutzername und Passwort für deinen IPTV Zugang. Wir zeigen dir, wie du ...
- 9 kudos
- 264 Views
- 6 replies
- 7 kudos
How can I configure Lakeflow Connect SQL Server CDC Gateway to use a desired VM type?
Hi Team,I'm evaluating Databricks Lakeflow Connect for SQL Server CDC ingestion and have run into a gateway provisioning issue.EnvironmentRegion: Australia EastSource: Azure SQL DatabaseCDC enabled successfully on the database and source tableSQL Ser...
- 264 Views
- 6 replies
- 7 kudos
- 7 kudos
Hi @Mado As of now, Lakeflow Connect's CDC Gateway VM types (driver/worker SKUs) are not configurable at the pipeline or connection level — they are managed internally by the platform. If the default SKU (Standard_E4d_v4) is unavailable in your regio...
- 7 kudos
- 104 Views
- 3 replies
- 1 kudos
Lakeflow connect SQL server ingestion can be made elastic?
Hi Everyone, One of our big ingestion tasks is lakeflow connect CDC ingestion of ERP data from a SQL server database. I am deploying the pipelines and jobs for this via DABs. We are ingesting about 80 tables from the database, a handful of which are ...
- 104 Views
- 3 replies
- 1 kudos
- 1 kudos
If you need 15 minute CDC interval - Migrate those specific jobs to the integrated CDC pipeline in continuous speed-optimized mode. If the 15 minute jobs involves fewer than 50 tables, it works directly. If not, split them into multiple pipelines. Mo...
- 1 kudos
- 111 Views
- 2 replies
- 0 kudos
1.Not able to access interent (gcp databricks) 2.both ingress and egress access given
1.Not able to access interent (gcp databricks) 2.both ingress and egress access given in default policy2.How to overcome this issue and i am using serverless3.in free databricks able to use interent in gcp based databricks it is creating issueimport...
- 111 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Hariharan_0510 , I took a look at both internal and external documentation and here is what I found. Building on @balajij8's reply, the serverless egress policy is the right place to start, but two things before you go hunting further. First, ...
- 0 kudos
- 130 Views
- 2 replies
- 2 kudos
Implementing a "Zero-Bus" Architecture in a Lakehouse: Best Practices for Shared Dimensions?
Hi Everyone,I am currently architecting a Medallion Lakehouse and moving toward a "Zero-Bus" (Bus Architecture) approach. My goal is to ensure that our core dimensions—such as Customer, Product, and Geography—remain consistent and conformed across al...
- 130 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,Great question. This is where most Medallion implementations either scale or get messy. Here's the pattern that has worked well, based on Kimball conformed dimensions mapped onto Unity Catalog.Quick note on naming: in Databricks, "Zerobus" is a pr...
- 2 kudos
- 2055 Views
- 4 replies
- 0 kudos
Resolved! Slow writes to managed volume
Hi All, I am using managed volumes as an intermediate storage to write a decrypted file before moving to data lake storage. Strangely the write operation is taking a lot of time (22 mins) to write a small file to volumes and it takes only few seconds...
- 2055 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi All, Thank you for you advice regarding this issue, but the root cause was something else. We had 2 private endpoints for ADLS (BLOB and DFS), and the firewall was only open for one BLOB PE. As a process Databricks first tries to resolve connectio...
- 0 kudos
- 185 Views
- 3 replies
- 1 kudos
Databricks AI/BI Get Session User
Hello everyone,I am currently working with Databricks AI/BI Dashboards and I have a specific requirement regarding user context.I need to dynamically capture the "session user" (the person who is currently viewing the dashboard) rather than the "exec...
- 185 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi,Short answer: No. There's no built-in function or parameter that returns the viewer's identity while a dashboard runs on the publisher's credentials.Why: When you publish with Share data permissions (the default, formerly called "Embed credentials...
- 1 kudos
- 297 Views
- 6 replies
- 5 kudos
Resolved! org.apache.iceberg.connect.IcebergSinkConnector to sink data from kafka to databricks - large volume
Hey.We have different number of topics per let's call it "subject". The connector works great for us if there are not so many topics (partitions) in subject. We have issue when we try to sink 1500 topics, 3 partitions each. We've sharded the topics i...
- 297 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello All.Thank you all for help. Here is what worked for us:## Original symptomsEach connector started normally and consumed records for more than an hour. Kafka Connect reportedthe connector and tasks as `RUNNING`, and source-topic lag decreased. T...
- 5 kudos
- 17193 Views
- 10 replies
- 8 kudos
The log in function and password reset function in the community edition do not work
I want to register a databrick account. I already set up my account. I also receive the email to set my password. However, I cannot use my password to log in the community account. I can use it to log in my standard account. I also click the reset th...
- 17193 Views
- 10 replies
- 8 kudos
- 8 kudos
Databricks guys claim at 190 Billion valuation ? or more ??Still struggling to send password reset email ??handover the project to me, I can create better customer experience!!! No team needed!!https://www.webassessor.com/#/twForgotPassword is the wo...
- 8 kudos
- 74 Views
- 0 replies
- 0 kudos
Built my first Databricks App with Lakebase
Screenshot of my deployed app attached. Thanks for the great quest!#Databricks #Lakebase #AppKit
- 74 Views
- 0 replies
- 0 kudos
- 100 Views
- 1 replies
- 0 kudos
Tip: Why your Delta MERGE fails with "multiple source rows matched" (and the simple fix)
Hi everyone If you run incremental loads with MERGE INTO on Delta tables, you may have seen a merge fail with an error saying multiple source rows matched the same target row.Why it happensA MERGE compares each row in your source batch to your target...
- 100 Views
- 1 replies
- 0 kudos
- 0 kudos
Great breakdown! That exact error has tripped up many a CDC pipeline. When working in PySpark, I typically use a similar window approach with row_number() and a strict tiebreaker like a sequence number before registering the temporary view for the me...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
