Databricks Community

Khasim_1

Hi everyone,I am currently refining the architecture for an end-to-end Lakehouse project and moving toward a "Zero-Bus" (Bus Architecture) model. My primary objective is to enforce dimension conformity (Customer, Product, Geography) across multiple i...

Khasim_1

Hi Everyone,I am currently architecting a Medallion Lakehouse and moving toward a "Zero-Bus" (Bus Architecture) approach. My goal is to ensure that our core dimensions—such as Customer, Product, and Geography—remain consistent and conformed across al...

Khasim_1

Hi Everyone,Recently, I led an end-to-end Lakehouse project for a client that was struggling with manual, error-prone reporting and disconnected data silos. Their core business problem was simple yet critical: the business could not trust its own num...

Khasim_1

Hi everyone,I’m currently evaluating the role of the AI Gateway in a Medallion Architecture implementation. While the ability to centralize rate limiting and credential management is a clear win for production stability, I’m interested in how others ...

Khasim_1

Hi everyone,I’m currently evaluating Lakeflow Connect for our ingestion layer. While the setup for replicating source databases into the Lakehouse is remarkably streamlined, I’m looking for operational best practices from those of you using it in pro...

Khasim_1

Hi @Islam_hoti ,You have hit the classic 'Streaming vs. Mutation' wall. readStream on a Delta table assumes an append-only sequence; once an UPDATE happens, the commit versioning sequence is invalidated, hence the loud failure.Here is how we reasoned...

Khasim_1

Hi @Islam_hoti ,Photon is fantastic, but it’s an 'all-or-nothing' value proposition. Once you hit a fallback, you lose the vectorized execution benefit for that entire sub-tree, and you're still paying the DBU premium. Here is how we approach auditin...

Khasim_1

Hi @Islam_hoti ,This is the 'Silent Architect' dilemma—when the standard tool is technically suboptimal but organizationally necessary. We’ve wrestled with this, and here’s where we landed:The 'Standardization Premium' is real: We reasoned that the c...

Khasim_1

Hi @Islam_hoti , Great question! We’ve faced a similar scenario with a high-volume transactional table (though ours is roughly 3TB).MERGE vs. Clustering: When we moved from date-partitioning to Liquid Clustering, our MERGE performance improved signif...

Khasim_1

Great guide, Arthur! I particularly agree with your point on defining table grain and relationships—those are often the silent killers of 'Genie' accuracy, especially when joins inflate totals. In my current work with complex medallion architectures,...

Databricks Community

User Stats

User Activity

Implementing a "Zero-Bus" Architecture with Unity Catalog

Implementing a "Zero-Bus" Architecture in a Lakehouse: Best Practices for Shared Dimensions?

Architecting a Medallion Lakehouse

Balancing Model Agility and Centralized Governance with Mosaic AI Gateway

Operationalizing Lakeflow Connect:Handling Upstream Schema Evolution & Historical Backfill

Re: Streaming read fails with "Detected a data update" after a restatement job touches the

Re: Photon enabled but a large share of the plan is falling back, cost up and runtime flat

Re: At what data size do you stop reaching for Spark?

Re: Liquid clustering vs. partitioning for a 5 TB Silver table with frequent MERGEs

Re: Configuring Genie: practical steps for more reliable answers