csv file processing

[vadi]

whats best possible solution to process csv file in databricks.

Please consider scalability,optimization, qa give m best solution...

[szymon_dybczak]

Hi @vadi

In my opinion, the best way is to use autoloader. For performance reason, it's also beneficial to provide schema upfront. Assigning schema manually also will improve the performance because it does not do the schema inference with the huge set of files. Thus, performance will be enhanced. 

https://medium.com/@rahuljax26/autoloader-cookbook-part-1-d8b658268345

[Kaniz_Fatma]

Hi @vadi, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!