Databricks Community

Mridu · 03-14-2026

Hi Databricks Community I’m working on designing a practical Entity Resolution / Deduplication framework on Databricks, and I wanted to share a high‑level blueprint and learn from others who’ve tackled similar problems at scale.The use case is fairl...

Mridu · 06-22-2026

You’re right — this is a real gap today in Lakeflow / pipeline orchestration.The current model is effectively “pipeline = unit of refresh”, not “table/view = unit of refresh with policy”, which makes mixed-frequency refresh use cases awkward.What peo...

Mridu · 03-15-2026

Hi @Ashwin_DSA -Thanks again for the detailed accelerator mapping — one follow‑up question I’m curious about from a delivery perspective:In regulated BFSI projects, how do teams typically decide where to draw the line between accelerator reuse vs cus...

Mridu · 03-15-2026

Hi @Ashwin_DSA — thank you for the detailed and thoughtful response, this is extremely helpful.I completely agree with your point about not reinventing the wheel and leveraging the Databricks Entity Resolution Solution Accelerators as a starting poin...

Mridu · 03-14-2026

This is expected behavior in Spark Structured Streaming, and the key point is that file streaming is not just driven by the checkpoint.Spark uses file metadata tracking at the source level, not only checkpoint state, to decide whether a file is “new”...

Databricks Community

User Stats

User Activity

Blueprint: Entity Resolution(Dedup & Golden Records)on Databricks — blocking, fuzzy scoring, MLflow

Re: Adhoc Table Refresh in Lakeflow Spark Declarative Pipelines (SDP)

Re: Blueprint: Entity Resolution(Dedup & Golden Records)on Databricks — blocking, fuzzy scoring,

Re: Blueprint: Entity Resolution(Dedup & Golden Records)on Databricks — blocking, fuzzy scoring,

Re: Spark Streaming – Old file not processed with new checkpoint and new output path