Databricks Community

yit

In all Databricks documentation, the examples use import dlt to create streaming tables and views. But, when generating sample Python code in ETL pipeline, the import in the sample is:import pyspark import pipelines as dpWhich one is the correct libr...

yit

Hey everyone,We’ve been using Autoloader extensively for a while, and now we’re looking to transition to full Lakeflow Declarative Pipelines. From what I’ve researched, the reader part seems straightforward and clear.For the writer, I understand that...

yit

Hey everyone,I want to set column type as empty struct via schema hints without specifying subfields. Then I expect the struct to be evolved with subfields through schema evolution when new subfields appear in the data. But, I've found in the documen...

yit

We are currently implementing Autoloader for JSON files with nested struct fields. The goal is to detect the fields as structs, and to have schema evolution.The schema evolution mode is set to addNewColumns, and inferColumnTypes option is set to true...

yit · 09-02-2025

Hey everyone,I’m trying to clarify a confusion in AutoLoader regarding trigger batches and micro-batches when using .forEachBatch.Here’s what I understand so far:Trigger batch – Controlled by cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTr...

yit

@K_Anudeep thank you for the reply! This is how I've developed it, but I have some erroneous files where that exact column is array instead of struct, so it's inferred as string (the most generic type between array and struct).My goal was to define v...

yit

It is possible to define empty struct as column type through schema hints, but it won't do schema evolution if subfields appear in the data for that column. Conclusion when working with JSON files and 'addNewColumns' as schema evolution mode: You can...

yit

@szymon_dybczak I did not conduct memory efficiency benchmarking, as it's not the main evaluation we seek for. Considering the current scenario, we definitely want to use struct.The problem that should be solved is how to define an empty struct in sc...

yit

@szymon_dybczak Struct has its own benefits over variant. It's more memory efficient, and it shows the full nested schema in Overview in Unity Catalog, while the variant type just shows the name of the first level field. Basically you can navigate th...

yit

@szymon_dybczak Not if you set the option inferColumnTypes to true.

Databricks Community

User Stats

User Activity

Difference between libraries dlt and dp

How to implement MERGE operations in Lakeflow Declarative Pipelines

Schema hints: define column type as struct and incrementally add fields with schema evolution

Autoloader: struct field inferred as string

Autoloader: Trigger batch vs micro-batch (as in .forEachBatch)

Re: Schema hints: define column type as struct and incrementally add fields with schema evolution

Re: Autoloader: struct field inferred as string

Re: Autoloader: struct field inferred as string

Re: Autoloader: struct field inferred as string

Re: Autoloader: struct field inferred as string