Hi @SergioSchena ,

thanks for sharing it, you solution looks pretty solid and I wanted to implement it but copying part of CSV inclemently was making me unable to sleep 😋

I just wonder if there is better (easier) option we can setup. I am currently testing various option and I spotted that Incremental Folder Update Structure is possible without Synapse Workspace - just CSVs on ADLS storage.
Then the data lands in following pattern (example with 60 min incremental) :

conatiner
--2024-10-15T10.00.00Z
----Microsoft.Athena.TrickleFeedService
----OptionsetMetadata
----model.json
----Table_A
------2020.csv
------2021.csv
----Table_B
------2021.csv
------2022.csv
----Table_C
--2024-10-15T11.00.00Z
--2024-10-15T12.00.00Z

Then we could potentially run autoloader for each tables with pattern "root_path/*/Table_A/*.csv.

In this option we could avoid reading parts of CSV and copying to other storage.
What do you think about this?