when attempting to load a large 800 MB CSV file

[violeta482yee]

Hello everyone,

I’m facing an issue when attempting to load a large 800 MB CSV file from ADLS using Auto Loader. Unfortunately, the notebook crashes during the loading process. Has anyone experienced something similar or have any suggestions on how to handle this?

Any tips to resolve or troubleshoot this issue would be greatly appreciated.

Thanks in advance,

Violeta

[Alberto_Umana]

Hi @violeta482yee,

Have you checked the resource availability of the compute attached to the notebook?

One solution could be: to usecloudFiles.maxBytesPerTriggerOption: This option allows you to control the maximum number of bytes processed in each trigger. For example, setting it to 100m will process up to 100 MB of data per trigger. This can help prevent the notebook from crashing due to memory overload

But it requires more investigation on the compute being used to determine the reason.