- 1694 Views
- 4 replies
- 0 kudos
Resolved! Databricks Cost Estimation Template
Hi Databricks Team, Is there a standard Databricks cost estimation template(xl), sizing calculator, or TCO tool that allows us to provide the following inputs and derive an approximate monthly and annual platform cost:Source systems and their types (...
- 1694 Views
- 4 replies
- 0 kudos
- 0 kudos
@Phani1 @robert7878 - You can build a AI/BI dashboard pretty easily using Genie, just give what is required and how you want it to be represented.
- 0 kudos
- 393 Views
- 2 replies
- 1 kudos
Add resource to app space
Hi,I recently started trying out the Genie App Builder first presented at DAIS. For this I created an app space and I wanted to connect my app to a lakebase project. But I can't seem to find any options to do this like it works with apps which are no...
- 393 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @JanW Following up on the answer above from @saurabh18cs and the results of a quick API test. Thank you for setting out what you tried and how you compared it with an app outside a space. That made the case easy to reproduce Short answer: You...
- 1 kudos
- 129 Views
- 1 replies
- 0 kudos
Managed table conversion and vacuum
Hello everyone,I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE <table_name> SET MANAGEDDatabricks document...
- 129 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Davide, this thread never got a reply, but you asked the same thing in SET MANAGED and rollback in relation to VACUUM and posted your test result there, so I'm linking it for anyone who lands here first.Short version: the 14-day rollback window i...
- 0 kudos
- 1153 Views
- 5 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 1153 Views
- 5 replies
- 0 kudos
- 617 Views
- 3 replies
- 3 kudos
Resolved! Best practices and optimization tips for Delta Lake tables
Hi everyone,I’m currently working with Delta Lake and wanted to start a discussion around performance tuning, maintenance, and best practices.I’m interested in hearing how others in the community approach a few key areas:Maintenance Operations: How o...
- 617 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Khasim_1 Maintenance operations: OPTIMIZE and VACUUMThe frequency of maintenance operations should depend on the table’s ingestion pattern, update/delete volume, file sizes, query performance, and storage cost. Avoid running them on a fixed sch...
- 3 kudos
- 3261 Views
- 3 replies
- 7 kudos
How to Unit Test Spark Declarative Pipelines
Every engineer who ships application code writes unit tests for it. Yet the data pipeline that feeds the same application is often left with a manual run and a dashboard that someone eventually notices is wrong. The gap has been more about tooling th...
- 3261 Views
- 3 replies
- 7 kudos
- 7 kudos
Thanks for this, the quickstart was a great starting point. The per-test isolation with the redirecting_* schemas is really nice: we could test AUTO CDC edge cases (deletes, out-of-order updates, null keys) on mock data, without touching real tables....
- 7 kudos
- 767 Views
- 3 replies
- 2 kudos
Building a Visitor Data Pipeline for Digital Membership Card
Hi everyone, I'm working on a project where we collect visitor activity from multiple touchpoints such as ticketing, membership sign-ups, event participation, mobile app interactions, and digital membership card usage. The goal is to create a unified...
- 767 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello @dixcyscott Streaming for check-ins, batch for slower sources, and a gold profile built from the event facts. Ingestion and the silver tables run in one Lakeflow pipeline, with a Databricks SQL materialized view on top for near real-time. Q1. ...
- 2 kudos
- 631 Views
- 4 replies
- 4 kudos
Resolved! Materialized View with Incremental Refresh - Change Data Feed required on source tables?
Question:To create a materialized view with incremental refresh, is the Delta Lake feature Change Data Feed required on all of the source tables that the materialized view consumes? Or, is it not required for the creation of the materialized view, bu...
- 631 Views
- 4 replies
- 4 kudos
- 4 kudos
@johnb1 - Hello Johnb1 - CDF is not required, either to create the materialized view or to refresh it incrementally. The current Databricks docs (updated October 2026) list it only as a recommendation for better performance.Row tracking is what can ...
- 4 kudos
- 1225 Views
- 6 replies
- 7 kudos
Resolved! Unity Catalog v.2.0?
Are there plans for a v.2.0 of Unity Catalog? I find the organization of tables in databricks to be very constrained and inflexible. Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...
- 1225 Views
- 6 replies
- 7 kudos
- 7 kudos
Fair points, especially for data coming from systems with richer naming. I can't speak to the roadmap, but here's what works for us today:Aliases: Create views in a consumer-facing catalog, like reporting.sales.outside_sales, that point to the real t...
- 7 kudos
- 1642 Views
- 14 replies
- 12 kudos
Resolved! Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 1642 Views
- 14 replies
- 12 kudos
- 12 kudos
Hello @ankit2604, I took a look at both internal and external documentation and here is what I found. Thanks @balajij8, @Satyasai, @Jen_Evangelista and @data_pulse for doing most of the work here. The shape they've described is right: streaming table...
- 12 kudos
- 575 Views
- 3 replies
- 4 kudos
Resolved! Zerobus rescue (VARIANT) not working. No Preview Available.
Has anyone tried using the rescue feature for json in zerobus-ingest? It seems like a very significant feature, assuming the schema incoming json is even 1% outside the control of team that may be ingesting on the databricks side. There needs to be ...
- 575 Views
- 3 replies
- 4 kudos
- 4 kudos
You’re reading the intended behavior correctly. Zerobus should route non-conforming fields into the rescue column instead of rejecting the record. Your error suggests the feature isn’t active on that table. Per the docs, if no column qualifies, the f...
- 4 kudos
- 1651 Views
- 10 replies
- 6 kudos
Understanding Parquet File Storage for Large Datasets
Hi everyone,I’m learning about Parquet files and how they are used in Databricks for storing large datasets.I’m trying to understand how column-based storage works in a practical situation.For example, suppose an e-commerce company has 500 million or...
- 1651 Views
- 10 replies
- 6 kudos
- 784 Views
- 4 replies
- 2 kudos
How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?
We are using AUTO CDC (APPLY CHANGES INTO) to process change data capture events from an upstream source (via Autoloader ingesting CDC files landed in cloud storage) into our Silver layer tables. Our source occasionally delivers out-of-order events —...
- 784 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello @Khasim_1, I took a look at both internal and external documentation and here is what I found. You're in good hands in this thread already. @Brahmareddy's STRUCT suggestion and @data_pulse's test both line up with the documented behavior, and @...
- 2 kudos
- 506 Views
- 1 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 506 Views
- 1 replies
- 0 kudos
- 0 kudos
Nice write-up, the read_files plus mergeSchema point is one I'd have missed. A few things from the current docs that may help with edges 2 and 4, since some of it has changed recently.On expectations: the incremental refresh page now says materialize...
- 0 kudos
- 715 Views
- 3 replies
- 2 kudos
Implementing a "Zero-Bus" Architecture in a Lakehouse: Best Practices for Shared Dimensions?
Hi Everyone,I am currently architecting a Medallion Lakehouse and moving toward a "Zero-Bus" (Bus Architecture) approach. My goal is to ensure that our core dimensions—such as Customer, Product, and Geography—remain consistent and conformed across al...
- 715 Views
- 3 replies
- 2 kudos
- 2 kudos
Short answer: I'd build the golden record in a dedicated conformed layer that sits between Silver and the marts, rather than before Silver or inside any one mart. Treat that layer as a governed data product with a stable, versioned contract, and let...
- 2 kudos