topic Re: csv file processing in Data Engineering

csv file processing

vadi — Tue, 06 Aug 2024 16:14:06 GMT

whats best possible solution to process csv file in databricks.

Please consider scalability,optimization, qa give m best solution...

Re: csv file processing

szymon_dybczak — Tue, 06 Aug 2024 20:17:37 GMT

In my opinion, the best way is to use autoloader. For performance reason, it's also beneficial to provide schema upfront. Assigning schema manually also will improve the performance because it does not do the schema inference with the huge set of files. Thus, performance will be enhanced.

https://medium.com/@rahuljax26/autoloader-cookbook-part-1-d8b658268345

Re: csv file processing

Retired_mod — Thu, 08 Aug 2024 06:39:09 GMT

Hi @vadi, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!