Databricks Community

aleksandra_ch

How to: Master Streaming Tables and Materialized Views Make sure to check out the previous post: https://community.databricks.com/t5/technical-blog/spark-declarative-pipelines-how-to-series-part-1-how-to-save/ba-p/149180 Welcome to the second post of...

aleksandra_ch · 02-25-2026

Spark Declarative Pipelines “How-To” Series Introduction Lakeflow Spark Declarative Pipelines (SDP) is a framework designed for building scalable, maintainable ETL pipelines. Everyone should want to make the transition from traditional imperative ETL...

aleksandra_ch

Hi @tsam , Can you share few details: Which DBR is the job on?How many DEEP CLONEs you need to run in total?What is the parallelism of the for-each task?Are the cloned tables optimized (e.g. there is no "small file problem")?Can you share the Heap Hi...

aleksandra_ch

Hi @databrick_enthu , The error is unfortunately misleading. The Materialized View / Streaming Table on Serverless Generic Compute is not yet available for a Preview. For now, you have two possibilities: Attach the notebook to a Serverless SQL Wareh...

aleksandra_ch

HI @nb92 , Only Scalar Python UDFs are allowed for on-demand feature computation. This page provides the recommended approach. Best regards,

aleksandra_ch

Hi @Steffen , Perhaps you can classify ids by their data frequency, e.g. low, med, and high-frequency, and process them in different jobs? This will avoid skewness. Also as a quick test I would suggest to try to run it on serverless - it will scale ...

aleksandra_ch

Hi @antoalphi , You are correct when saying that Materialized View's logic can be implemented using a regular Delta table (outside of SDP). What Materialized Views bring you: Automatic lineage: with regular Delta table you have to provide dependencie...

Databricks Community

User Stats

User Activity

SDP “How-To” Series. Part 2: How to Master Streaming Tables and Materialized Views

Spark Declarative Pipelines “How-To” Series. Part 1: How to Save Results Into A Table

Re: Driver memory utilization grows continuously during job

Re: Cannot create streaming table

Re: Recommended Python UDFs for On-Demand Feature Computation in Databricks

Re: Partition optimization strategy for task that massively inflate size of dataframe

Re: Spark Declarative Pipelines “How-To” Series. Part 1: How to Save Results Into A Table