- 250 Views
- 6 replies
- 10 kudos
Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 250 Views
- 6 replies
- 10 kudos
- 10 kudos
Great points from @balajij8 on the SDP/LDP approach. I've worked on a few Data Vault implementations, and these are the things that tend to bite when the Business Vault grows:1. Hash keys: standardize before hashing. Trim, upper-case and use a fixed ...
- 10 kudos
- 162 Views
- 3 replies
- 4 kudos
Resolved! Zerobus rescue (VARIANT) not working. No Preview Available.
Has anyone tried using the rescue feature for json in zerobus-ingest? It seems like a very significant feature, assuming the schema incoming json is even 1% outside the control of team that may be ingesting on the databricks side. There needs to be ...
- 162 Views
- 3 replies
- 4 kudos
- 4 kudos
You’re reading the intended behavior correctly. Zerobus should route non-conforming fields into the rescue column instead of rejecting the record. Your error suggests the feature isn’t active on that table. Per the docs, if no column qualifies, the f...
- 4 kudos
- 1112 Views
- 10 replies
- 6 kudos
Understanding Parquet File Storage for Large Datasets
Hi everyone,I’m learning about Parquet files and how they are used in Databricks for storing large datasets.I’m trying to understand how column-based storage works in a practical situation.For example, suppose an e-commerce company has 500 million or...
- 1112 Views
- 10 replies
- 6 kudos
- 496 Views
- 4 replies
- 2 kudos
How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?
We are using AUTO CDC (APPLY CHANGES INTO) to process change data capture events from an upstream source (via Autoloader ingesting CDC files landed in cloud storage) into our Silver layer tables. Our source occasionally delivers out-of-order events —...
- 496 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @Khasim_1, I took a look at both internal and external documentation and here is what I found. You're in good hands in this thread already. @Brahmareddy's STRUCT suggestion and @data_pulse's test both line up with the documented behavior, and @...
- 2 kudos
- 368 Views
- 1 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 368 Views
- 1 replies
- 0 kudos
- 0 kudos
Nice write-up, the read_files plus mergeSchema point is one I'd have missed. A few things from the current docs that may help with edges 2 and 4, since some of it has changed recently.On expectations: the incremental refresh page now says materialize...
- 0 kudos
- 1955 Views
- 0 replies
- 1 kudos
How to Unit Test Spark Declarative Pipelines
Every engineer who ships application code writes unit tests for it. Yet the data pipeline that feeds the same application is often left with a manual run and a dashboard that someone eventually notices is wrong. The gap has been more about tooling th...
- 1955 Views
- 0 replies
- 1 kudos
- 383 Views
- 3 replies
- 2 kudos
Implementing a "Zero-Bus" Architecture in a Lakehouse: Best Practices for Shared Dimensions?
Hi Everyone,I am currently architecting a Medallion Lakehouse and moving toward a "Zero-Bus" (Bus Architecture) approach. My goal is to ensure that our core dimensions—such as Customer, Product, and Geography—remain consistent and conformed across al...
- 383 Views
- 3 replies
- 2 kudos
- 2 kudos
Short answer: I'd build the golden record in a dedicated conformed layer that sits between Silver and the marts, rather than before Silver or inside any one mart. Treat that layer as a governed data product with a stable, versioned contract, and let...
- 2 kudos
- 372 Views
- 2 replies
- 1 kudos
Implementing a "Zero-Bus" Architecture with Unity Catalog
Hi everyone,I am currently refining the architecture for an end-to-end Lakehouse project and moving toward a "Zero-Bus" (Bus Architecture) model. My primary objective is to enforce dimension conformity (Customer, Product, Geography) across multiple i...
- 372 Views
- 2 replies
- 1 kudos
- 1 kudos
To implement a conformed dimension bus architecture in a Databricks Lakehouse, teams typically build master shared dimensions in a dedicated "Master Silver" layer—governed in a centralized Unity Catalog schema with strict row/column-level ACLs—while ...
- 1 kudos
- 3637 Views
- 5 replies
- 2 kudos
Python logger.info() not showing inside applyInPandas (but print() works) — why?
Problem: In Databricks, logs from an external binary (via os.system) show up, but Python logger.info() inside groupBy(...).applyInPandas(...) does not. print(..., flush=True) does show up.Why: applyInPandas runs your function as a pandas UDF. That c...
- 3637 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi Ben, The stdout logging-handler trick works on a classic/standard cluster, but on serverless it will not surface in the notebook cell, and that is expected rather than a problem with the helper. Here is what is going on. applyInPandas runs your Py...
- 2 kudos
- 4211 Views
- 8 replies
- 12 kudos
Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition
As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks F...
- 4211 Views
- 8 replies
- 12 kudos
- 12 kudos
@Brahmareddy I haven't used this exact pattern at multi-million-row scale, but I agree that the tradeoffs start to change as the dataset and query volume grow.I would probably keep Delta as the source of truth and separate the serving layer based on ...
- 12 kudos
- 818 Views
- 2 replies
- 3 kudos
Resolved! Designing an Effective "Quarantine" Pattern for Failed DLT Expectations
Hi everyone,I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.While expect_or_drop and expect_or_fail are straightforward, I’m struggling to fi...
- 818 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @Khasim_1, this was an interesting one and I wanted to experiment a bit. I put this together if it helps - https://medium.com/lab-notes/three-lakeflow-quarantine-designs-tested-and-the-one-condition-every-queue-needs-1bea1c1588ff
- 3 kudos
- 774 Views
- 6 replies
- 6 kudos
Best Architecture Approach for Databricks Multi-Cloud and Multi-Region Deployment
I am working on a data platform requirement where we need to support multiple cloud providers and multiple regions using Databricks.For example:AWS – Region 1AWS – Region 2Azure – Region 1Azure – Region 2The requirement is to maintain data processing...
- 774 Views
- 6 replies
- 6 kudos
- 6 kudos
@Srini_Pesala You can deploy one workspace per region per cloud provider. Unity Catalog enforces certain limits on metastore per region. Data residency is handled natively through Databricks Geos and it ensures customer data is strictly processed and...
- 6 kudos
- 764 Views
- 5 replies
- 5 kudos
Resolved! Unity Catalog v.2.0?
Are there plans for a v.2.0 of Unity Catalog? I find the organization of tables in databricks to be very constrained and inflexible. Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...
- 764 Views
- 5 replies
- 5 kudos
- 5 kudos
Fair points, especially for data coming from systems with richer naming. I can't speak to the roadmap, but here's what works for us today:Aliases: Create views in a consumer-facing catalog, like reporting.sales.outside_sales, that point to the real t...
- 5 kudos
- 628 Views
- 3 replies
- 6 kudos
Resolved! Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption
Hi everyone, we are defining our PII strategy in Unity Catalog. We are split on whether to: A) Redact/Hash PII at the Silver layer (permanent change), or B) Keep PII in Silver and use Dynamic Data Masking at the Gold/View layer.Does "Redact at Inges...
- 628 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi,In practice most teams end up with a hybrid: keep raw PII in one tightly restricted place, and use dynamic masking everywhere else.1. Does redacting at Silver (A) create friction? Yes, often. Once PII is hashed or dropped in Silver, the raw data i...
- 6 kudos
- 482 Views
- 2 replies
- 3 kudos
Resolved! Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle
Hi everyone,Hi everyone, I’m architecting a data-sharing program using Delta Sharing for external partners who do not have a Databricks workspace.How are others automating the recipient credential lifecycle? Do you have an automated process for rotat...
- 482 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,1. Credential lifecycle: If your partners have their own identity provider (Entra, Okta, etc.), consider OIDC token federation instead of bearer tokens. The partner's IdP issues short-lived tokens, so there's nothing for you to rotate or leak. It ...
- 3 kudos