When dealing with a large number of Parquet files (about 2500 in your case), the system might be running into resource limitations or timeouts. This can happen due to the sheer volume of data being processed at once.

The failure might be due to insufficient resources allocated to handle the large volume of data. Ensure that your cluster has enough resources (CPU, memory) to handle the load.