I am loading avro files into the detla tables. I am doing this for multiple tables and some files are big like (2-3GB) and most of them are small like in few MBs.
I am using autoloader to load the data into the delta tables.
My question is:
What is the best practice , do I infer the schema or provide the schema while using autoloader.
I understand that if I provide the schema , the processing would be faster however it would be time consuming to provide the schema for tables that has many columns.
Any suggestions?