when attempting to load a large 800 MB CSV file

violeta482yee — Thu, 16 Jan 2025 10:04:07 GMT

Hello everyone,

I’m facing an issue when attempting to load a large 800 MB CSV file from ADLS using Auto Loader. Unfortunately, the notebook crashes during the loading process. Has anyone experienced something similar or have any suggestions on how to handle this?

Any tips to resolve or troubleshoot this issue would be greatly appreciated.

Thanks in advance,

Violeta

Re: when attempting to load a large 800 MB CSV file

Alberto_Umana — Thu, 16 Jan 2025 12:35:18 GMT

Hi @violeta482yee,

Have you checked the resource availability of the compute attached to the notebook?

One solution could be: to usecloudFiles.maxBytesPerTriggerOption: This option allows you to control the maximum number of bytes processed in each trigger. For example, setting it to 100m will process up to 100 MB of data per trigger. This can help prevent the notebook from crashing due to memory overload

But it requires more investigation on the compute being used to determine the reason.

topic when attempting to load a large 800 MB CSV file in Databricks Free Edition Help

when attempting to load a large 800 MB CSV file

Re: when attempting to load a large 800 MB CSV file