Databricks Community

binlogreader

I am trying to settle a design default for our declarative pipelines, and I would like to hear how others decide between streaming tables and materialized views when a table could be built either way.When I started with streaming tables, I thought th...

binlogreader

Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.Setup: A continuous DLT pipeline on classic...

binlogreader

Hi All, I'm trying to solve two related problems on our lakehouse and I keep arriving at data contracts as the answer, but I can't find much on what a first working version looks like in practice on Databricks:(a) Upstream application teams change th...

binlogreader

Hi All, I need an up-to-date plain Delta copy of a streaming table created by Lakeflow pipeline. My first thought was CLONE, and that path is closed. The clone docs say streaming tables and materialized views cannot be the source or target of a deep ...

binlogreader

Hi All,I'm evaluating metadata-driven frameworks for Lakeflow Spark Declarative Pipelines, and I've found two that appear to serve the same purpose, both from Databricks organizations:- databrickslabs/dlt-meta, whose feature/sdp-meta branch renames t...

binlogreader

@swang-pg We did not run into the error but had a similar situation of stitching tables while using Salesforce and Mysql connectors so I can share what we did. Two things about the situation shape what the merge can look like.The first is that the me...

binlogreader

@Ajay-Pandey Are you trying to solve the cold start issue, or having continuously running cluster driven by business goals? It would be helpful to get some context here.

binlogreader

@emma_s @DoTA Thank you for your incredibly detailed notes! I am going to sit with them for a while to internalize, and then will respond back with some ideas or questions.

binlogreader

@dkushari Thanks for the detailed blog post. I am implementing the a similar architecture using duckdb to read from tables from data lake, and some questions.Setup: We run a read-only external query layer, DuckDB embedded in a Go service, over Unity ...

binlogreader

@keshavmonga22 Is there anyone in your team with Admin access that can grant you temporary access or can check the event log?

Databricks Community

User Stats

User Activity

Streaming tables vs materialized views in declarative pipelines: what actually decides the choice?

DLT driver GC pressure during a large initial hydration. Is a bigger driver the only lever?

Data contracts on Databricks: what does a working MVP actually look like?

How do you keep a plain Delta copy of a streaming table? CLONE is not supported on streaming tables

sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?

Re: Merging 2 versions of an SCD table created from a managed ingestion pipeline

Re: Databricks Job cluster for continuous run

Re: Data contracts on Databricks: what does a working MVP actually look like?

Re: Performing DML Operations on Unity Catalog Managed Tables from External Engines

Re: Pipeline still needs USE SCHEMA on an old schema it no longer writes to