Comment
Databricks Partner

The CSV side of the solutions has a lot of issues if there are changes in the source system. The CSV files will not have the headers. You have to create them from the JSON files. ( Like CDM format here : https://github.com/microsoft/CDM/blob/master/docs/schema/examples/OrdersProductsCustomersLinked/mode...

 

This creates heavy issues in case the columns change. Please consider using the Synapse based Spark pools to make them DELTA tables and read from there. The engineering efforts to productionise the CSV based format is not balanced.