Databricks Community

tech2cloud · 02-11-2023

tech2cloud · 02-11-2023

I have folder structure at source such as/transaction/date_=2023-01-20/hr_=02/tras01.csv/transaction/date_=2023-01-20/hr_=03/tras02.csvWhere 'date_' and 'hr_' are my partitions and present in the dataset as well. But the streamReader does not read th...

tech2cloud · 04-12-2023

I managed to get the problematic column by redirecting the error records.

Databricks Community

User Stats

User Activity

Databricks Autoloader simply does not return results if the schema provided has something wrong. Whereas it should provide some error. It Simply does not returns any rows making it difficult to debug.

Databricks Autoloader streamReader does not include the partition column as part of output.

Re: Databricks Autoloader simply does not return results if the schema provided has something wrong. Whereas it should provide some error. It Simply does not returns any rows making it difficult to debug.