Databricks Community

cdn_yyz_yul

Hi everyone,Trying the CDF on materialized view (declarative pipeline) feature that is currently in Beta.- Configuration:** Verified the MVs (materialized veiws) have: 'delta.enableRowTracking'SHOW TBLPROPERTIES my_mv ('delta.enableRowTracking'); re...

cdn_yyz_yul · 06-04-2026

Hi everyone,- the situation:I have a Declarative pipeline. The pipeline consists several .py files.file1.py: creates a Materialized View: description.file2.py: create 2nd Materialized View by reading a table "transactions" and reading the MV "descri...

cdn_yyz_yul · 03-13-2026

Hello,Bronze: use classic or job compute, Autoloader with.option("mergeSchema", "true"). Schema evolution works correctly. data goes to bronze.my_bronze_table.Silver: uses serverless compute, reader reads bronze.my_bronze_table, does all necessary t...

cdn_yyz_yul · 01-21-2026

Is the following type of union safe with spark structured streaming?union multiple streaming dataframes, and each from a different source.Anything better solution ?for example, df1 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table1") ...

cdn_yyz_yul · 12-02-2025

With autoloader.option("cloudFiles.schemaEvolutionMode", "addNewColumns") I have done retry after getting org.apache.spark.sql.catalyst.util.UnknownFieldException: [UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during par...

cdn_yyz_yul

The feature that will help the most is the CDF on MV that is in beta right now. I have been testing it to get ready at my side and take advantage of it once it is in General Availability. The Automatic change data feed on Streaming tables would be us...

cdn_yyz_yul

The following is a test with Streaming delta table using a notebook. No MV and DLT involved. Contrast to the most recently databricks docs, readChangeFeed only starts working after "ALTER TABLE <table_name> SET TBLPROPERTIES (delta.enableChangeDataFe...

cdn_yyz_yul

Regarding stream reader:Streaming, which is what the docs recommend anyway: spark.readStream.option("readChangeFeed", "true").table("my_catalog.my_schema.my_mv") with no starting version. It takes the current snapshot as inserts and tracks changes fo...

cdn_yyz_yul

Good morning @Louis_Frolio Re: > My suspicion is the backfilled versions don't carry the row-tracking metadata that automatic CDF needs, and 35 might.1) SELECT * FROM table_changes('my_catalog.my_schema.my_mv', <an earlier enought timestamp that allo...

cdn_yyz_yul

Update: Introduce row level changes to my_mv. Before adding new data to my_mv, CDF has enabled, the error can be found in my previous message.At this stage:- added new rows to the delta table where my_mv gets its input.- run DLT pipeline, not selecti...

Databricks Community

User Stats

User Activity

Change data feed from a materialized view

Declarative pipeline full table refresh generates empty MV.

schema evolution with structured streaming: upstream schema change causes downstream writer fails.

unionbyname several streaming dataframes of different sources

how to avoid extra column after retry upon UnknownFieldException

Re: Change data feed from a materialized view

Re: Change data feed from a materialized view

Re: Change data feed from a materialized view

Re: Change data feed from a materialized view

Re: Change data feed from a materialized view