Databricks Community

cdn_yyz_yul

Hi everyone,- the situation:I have a Declarative pipeline. The pipeline consists several .py files.file1.py: creates a Materialized View: description.file2.py: create 2nd Materialized View by reading a table "transactions" and reading the MV "descri...

cdn_yyz_yul · 03-13-2026

Hello,Bronze: use classic or job compute, Autoloader with.option("mergeSchema", "true"). Schema evolution works correctly. data goes to bronze.my_bronze_table.Silver: uses serverless compute, reader reads bronze.my_bronze_table, does all necessary t...

cdn_yyz_yul · 01-21-2026

Is the following type of union safe with spark structured streaming?union multiple streaming dataframes, and each from a different source.Anything better solution ?for example, df1 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table1") ...

cdn_yyz_yul · 12-02-2025

With autoloader.option("cloudFiles.schemaEvolutionMode", "addNewColumns") I have done retry after getting org.apache.spark.sql.catalyst.util.UnknownFieldException: [UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during par...

cdn_yyz_yul · 11-10-2025

Hello everyone,In our implementation of Medallion Architecture, we want to stream changes with spark structured streaming. I would like some advice on how to use delta table as source correctly, and if there is performance (memory usage) concern in t...

cdn_yyz_yul · 03-17-2026

Thanks @Ashwin_DSA Continue the topic, given the context discussed, when there is stateful operations in silver, the mismatch is definitely in the checkpointed state, not in table schema.I further tested it by removing the join (which is the single s...

cdn_yyz_yul · 03-16-2026

The first option, setting the bronze table to use rescue. In silver layer, - read the table from bronzeprocessed the json in _rescued_dataadded rescued columns as new columns to the dataframe in silver, e.g., silver_df1.I do need to join (silver_df1...

cdn_yyz_yul · 03-15-2026

Thanks @Ashwin_DSA I started to test: changing the bronze layer from addNewColumns to rescure (schemaEvolutionMode) while still keeping the format setting of autoloader as.csv. Then processing the _rescued data at silver. Or, the last resort, having ...

cdn_yyz_yul · 03-13-2026

severless compute does not support spark.conf.set("spark.sql.streaming.stateStore.stateSchemaCheck", "false"), If using classic compute, this setting will disable strict schema validation.I am looking for a solution that allows us to use serverless a...

cdn_yyz_yul · 03-11-2026

In the case of checkpoint rest (delelte checkpoint but keeping delta table), my understanding is that (foreachBatch with a MERGE INTO) is the only solution to avoid duplicates in the delta table. for example, ------- df = df1a.unionByName(df2a).unio...

Databricks Community

User Stats

User Activity

Declarative pipeline full table refresh generates empty MV.

schema evolution with structured streaming: upstream schema change causes downstream writer fails.

unionbyname several streaming dataframes of different sources

how to avoid extra column after retry upon UnknownFieldException

delta as streaming source, can the reader reads only newly appended rows?

Re: schema evolution with structured streaming: upstream schema change causes downstream writer fail

Re: schema evolution with structured streaming: upstream schema change causes downstream writer fail

Re: schema evolution with structured streaming: upstream schema change causes downstream writer fail

Re: schema evolution with structured streaming: upstream schema change causes downstream writer fail

Re: unionbyname several streaming dataframes of different sources