Vaibhavaher2025 - 

I recommend trying the following: 

1. Write logs from executors to persist storage insideprocess_partition.

2. Use mapPartitions instead offoreachPartition to return responses back to the driver as a Dataframe

3. Check executor logs in Spark UI under Executors -> Logs for anylogger.info outputs.

Remember, foreachPartition doesn’t return data, so to see API responses, either write to external storage or use mapPartitions to collect results.