Databricks Community

rcostanza

In a DLT pipeline I have a bronze table that ingest files using Autoloader, and a derived silver table that, for this example, just stores the number of rows for each file ingested into bronze. The basic code example: import dlt from pyspark.sql impo...

rcostanza

What I'm trying to achieve: ingest files into bronze tables with Autoloader, then produce Kafka messages for each file ingested using a DLT sink.The issue: latency between file ingested and message produced get exponentially higher the more tables ar...

rcostanza · 07-29-2025

I have a small (under 20 tables, all streaming) DLT pipeline running in triggered mode, scheduled every 15min during the workday. For development I've set `pipelines.clusterShutdown.delay` to avoid having to start a cluster every update.I've noticed...

rcostanza · 07-10-2025

In the pricing page for Lakeflow Declarative Pipelines (formerly DLT), for serverless it shows a single cost of $0.35/DBU for both standard and performance optimized. But in the feature table below, it says standard is "Up to 70% cheaper than running...

rcostanza · 06-17-2025

I have a notebook where at the beginning I load several dataframes and cache them using localCheckpoint(). I run this notebook using an all-purpose cluster with autoscaling enabled, with a mininum of 1 worker and maximum 2.The cluster often autoscale...

rcostanza

Gotcha. Thanks for the reply.We already have a medallion architecture going in production for a while, based on DLT pipelines having most bronze/silver tables, with some separate jobs for silver tables that are more complex to materialize. It works, ...

rcostanza

The files are actually being ingested into bronze within seconds of being created, at least in low volumes - one file per table, created simultaneously, small amount of rows. I haven't tweaked anything in particular, so maybe file notification was al...

rcostanza

Thanks for your replyThe state store in theory should be at a very modest size, considering there's are at most 5 test files for each table, and that each have a very low amount of rows each (100 or so). In any case, in one of my tests I've tried to ...

rcostanza · 11-01-2024

Thanks for the replyI tried checking the link you provided but it seems it's a restricted system. Is there another place I can upvote it?

Databricks Community

User Stats

User Activity

Stateless streaming with aggregations on a DLT/Lakeflow pipeline

Trying to reduce latency on DLT pipelines with Autoloader and derived tables

Lakeflow pipeline (formerly DLT pipeline) performance progressively degrades on a persistent cluster

Serverless costs, standard vs performance optimized

DataFrame.localCheckpoint() and cluster autoscaling at odds with each other

Re: Stateless streaming with aggregations on a DLT/Lakeflow pipeline

Re: Trying to reduce latency on DLT pipelines with Autoloader and derived tables

Re: Trying to reduce latency on DLT pipelines with Autoloader and derived tables

Re: Changing git's author field when committing through Databricks