Read and process large CSV files that updates regularly

Kjetil — Tue, 25 Jun 2024 08:09:16 GMT

I've got a lot of large CSV files (> 1 GB) that updates regularly (stored in Data Lake Gen 2). The task is to concatenate these files into a single dataframe that is written to parquet format. However, since these files updates very often I get a read error. I've tested with both batch and streaming (autoloader). I think perhaps the only way to deal with this is to create a copy (snapshot) of the files, then process in batch. However that takes a very long time - ideally I would like to avoid that extra step if possible.

I've been stuck with this issue for two days now, so any help here is much appreciated.

Re: Read and process large CSV files that updates regularly

daniel_sahal — Tue, 25 Jun 2024 11:14:11 GMT

@Kjetil Since they are getting updated often then IMO making a copy would make sense.

What you could try is to create Microsoft.Storage.BlobCreated event to replicate the .CSV into the secondary bucket.

However, best practice would be to have some kind of incremental approach on the source side - creating a new file instead of appending to the existing one.

topic Read and process large CSV files that updates regularly in Data Engineering

Read and process large CSV files that updates regularly

Re: Read and process large CSV files that updates regularly