Databricks Community

cdn_yyz_yul

With autoloader.option("cloudFiles.schemaEvolutionMode", "addNewColumns") I have done retry after getting org.apache.spark.sql.catalyst.util.UnknownFieldException: [UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during par...

cdn_yyz_yul

Hello everyone,In our implementation of Medallion Architecture, we want to stream changes with spark structured streaming. I would like some advice on how to use delta table as source correctly, and if there is performance (memory usage) concern in t...

cdn_yyz_yul · 02-03-2025

Hello,What are the benefits of not "registering" Raw data into Unity Catalog when the data in Raw will be in its original format, such as .csv, .json, .parquet, etc?An example scenario could be:Data arrives at Landing as .zip; The zip will be verifie...

cdn_yyz_yul

Even though the input is csv, it has indeed some rows mis-formatted.

cdn_yyz_yul

Hi @Hubert-Dudek the input is csv. readStream reads csv with .option("cloudFiles.inferColumnTypes", "true"). then, df.toDF( ) is called to rename the column name. The original csv header has space, that's why error message has "test 1_2 Prime". The r...

cdn_yyz_yul

This example helped me to deploy ETL pipelines as tasks in jobs to different workspaces.bundle-examples/lakeflow_pipelines_python at main · databricks/bundle-examples · GitHub

cdn_yyz_yul

Hello @mark_ott Thanks for the explanation. Very clear and constructive. Our design is leveraging structured streaming but for batch use, with availableNow and checkpointing.The scenario that I described occurs in Gold storage. After some reading and...

cdn_yyz_yul

Thanks @bianca_unifeye As the trigger is AvailableNow, each trigger starts with a new spark session. Then, When setting the following, "By default, the stream returns the latest snapshot of the table when the stream first starts as an INSERT and futu...

Databricks Community

User Stats

User Activity

how to avoid extra column after retry upon UnknownFieldException

delta as streaming source, can the reader reads only newly appended rows?

Shoud data in Raw /Bronze be in Catalog?

Re: how to avoid extra column after retry upon UnknownFieldException

Re: how to avoid extra column after retry upon UnknownFieldException

Re: Pipelines not included in Databricks Asset Bundles deployment

Re: delta as streaming source, can the reader reads only newly appended rows?

Re: delta as streaming source, can the reader reads only newly appended rows?