Databricks Community

yit

Hey everyone,I want to set column type as empty struct via schema hints without specifying subfields. Then I expect the struct to be evolved with subfields through schema evolution when new subfields appear in the data. But, I've found in the documen...

yit

We are currently implementing Autoloader for JSON files with nested struct fields. The goal is to detect the fields as structs, and to have schema evolution.The schema evolution mode is set to addNewColumns, and inferColumnTypes option is set to true...

yit

Hey everyone,I’m trying to clarify a confusion in AutoLoader regarding trigger batches and micro-batches when using .forEachBatch.Here’s what I understand so far:Trigger batch – Controlled by cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTr...

yit · 08-12-2025

I am using Autoloader to ingest source data into Bronze layer Delta tables. The source files are JSON, and I rely on schema inference along with schema evolution (using mode: addNewColumns). To handle errors triggered by schema updates in the stream,...

yit · 08-07-2025

Hi everyone,I’ve set up Databricks Autoloader to ingest data from ADLS into a Delta table. The table is defined as an external Delta table, with its location pointing to a path in ADLS.Here’s the flow I’m using:On the first run for a given data sourc...

yit

@K_Anudeep thank you for the reply! This is how I've developed it, but I have some erroneous files where that exact column is array instead of struct, so it's inferred as string (the most generic type between array and struct).My goal was to define v...

yit

It is possible to define empty struct as column type through schema hints, but it won't do schema evolution if subfields appear in the data for that column. Conclusion when working with JSON files and 'addNewColumns' as schema evolution mode: You can...

yit

@szymon_dybczak I did not conduct memory efficiency benchmarking, as it's not the main evaluation we seek for. Considering the current scenario, we definitely want to use struct.The problem that should be solved is how to define an empty struct in sc...

yit

@szymon_dybczak Struct has its own benefits over variant. It's more memory efficient, and it shows the full nested schema in Overview in Unity Catalog, while the variant type just shows the name of the first level field. Basically you can navigate th...

yit

@szymon_dybczak Not if you set the option inferColumnTypes to true.

Databricks Community

User Stats

User Activity

Schema hints: define column type as struct and incrementally add fields with schema evolution

Autoloader: struct field inferred as string

Autoloader: Trigger batch vs micro-batch (as in .forEachBatch)

Autoloader: Unexpected UnknownFieldException after streaming query termination

Autoloader fails when creating external Delta table in same notebook

Re: Schema hints: define column type as struct and incrementally add fields with schema evolution

Re: Autoloader: struct field inferred as string

Re: Autoloader: struct field inferred as string

Re: Autoloader: struct field inferred as string

Re: Autoloader: struct field inferred as string