Hi everyone,

I'm encountering an issue with my DLT pipeline that I haven't been able to resolve. My pipeline reads a single CSV file that is over 100 GB in size. This file is continuously updated throughout the day. When DLT attempts to read the file (which takes a few minutes), it fails with an 'underlying files have been updated' error. As a result, I have to read snapshots, and the pipeline reads the entire file each time instead of just the recently added rows.

What would you recommend in this scenario?

Please note that the file size and type are managed by a third party, so I can't make any changes on that front.