I have a Dataverse Synapse link set up to extract data into ADLS gen2. I am trying to connect ADLS gen2 as the data source to read the data files in Databricks. I have CDC enabled for CDM Data with the partition of Year and Month.
So, for example, if I am receiving 10 files for the table Product, the problem is the column size varies for the different csv received and there is only a single model.json for them. This is causing issues when reading the data with respect to column variation while reading schema.