It seems like the issue you're encountering with incomplete file downloads when using the foreach method and a UDF in Spark might be related to the number of partitions and how tasks are distributed across them. Here are a few points to consider:

1. Task Distribution and Partitioning: When you set the number of partitions to a value greater than the number of cores available on the worker, it can lead to inefficient task distribution. This might cause some tasks to complete almost immediately without fully processing the data, resulting in incomplete file downloads. By reducing the number of partitions to less than the number of cores, you ensure that each core is fully utilized, leading to more consistent task execution and complete file downloads.

2. Network and I/O Bound Operations: Downloading files over the network can be I/O bound, and having too many concurrent tasks might overwhelm the network or the I/O subsystem, leading to incomplete downloads. Reducing the number of partitions can help mitigate this bylimiting the number of concurrent download operations.

3. Error Handling and Retries: Ensure that your UDF includes proper error handling and retry logic for network operations. This can help recover from transient network issues that might cause incomplete downloads.

4. Resource Allocation: Check the resource allocation for your Spark job. Ensure that there are enough resources (CPU, memory, and network bandwidth) allocated to handle the download tasks efficiently.

5. Spark Configuration: You might want to experiment with different Spark configurations related to task scheduling and resource allocation to find the optimal settings for your workload.