Databricks Community

binlogreader

Hi All,If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with ...

binlogreader

I am trying to settle a design default for our declarative pipelines, and I would like to hear how others decide between streaming tables and materialized views when a table could be built either way.When I started with streaming tables, I thought th...

binlogreader · 07-25-2026

Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.Setup: A continuous DLT pipeline on classic...

binlogreader · 07-25-2026

Hi All, I'm trying to solve two related problems on our lakehouse and I keep arriving at data contracts as the answer, but I can't find much on what a first working version looks like in practice on Databricks:(a) Upstream application teams change th...

binlogreader · 07-22-2026

Hi All, I need an up-to-date plain Delta copy of a streaming table created by Lakeflow pipeline. My first thought was CLONE, and that path is closed. The clone docs say streaming tables and materialized views cannot be the source or target of a deep ...

binlogreader

@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:What we believed. The pipelines committed every few minutes, so each table's transactio...

binlogreader

@swang-pg We did not run into the error but had a similar situation of stitching tables while using Salesforce and Mysql connectors so I can share what we did. Two things about the situation shape what the merge can look like.The first is that the me...

binlogreader

@Ajay-Pandey Are you trying to solve the cold start issue, or having continuously running cluster driven by business goals? It would be helpful to get some context here.

binlogreader

@emma_s @DoTA Thank you for your incredibly detailed notes! I am going to sit with them for a while to internalize, and then will respond back with some ideas or questions.

binlogreader

@dkushari Thanks for the detailed blog post. I am implementing the a similar architecture using duckdb to read from tables from data lake, and some questions.Setup: We run a read-only external query layer, DuckDB embedded in a Go service, over Unity ...

Databricks Community

User Stats

User Activity

S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runtime?

Streaming tables vs materialized views in declarative pipelines: what actually decides the choice?

DLT driver GC pressure during a large initial hydration. Is a bigger driver the only lever?

Data contracts on Databricks: what does a working MVP actually look like?

How do you keep a plain Delta copy of a streaming table? CLONE is not supported on streaming tables

Re: S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runt

Re: Merging 2 versions of an SCD table created from a managed ingestion pipeline

Re: Databricks Job cluster for continuous run

Re: Data contracts on Databricks: what does a working MVP actually look like?

Re: Performing DML Operations on Unity Catalog Managed Tables from External Engines