How to get response from API call made via executor
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-02-2025 06:52 AM
Hi Guys,
I'm trying to call multiple APIs via executor using foreach partition, However as API response is getting returned at executor level I'm unable to see the response of API weather its 200 or 500.
I dont want my APIs to execute on driver so I'm avoiding toPandas or Multi threading on Driver side.
Note: I've enabled logs to save in the Volume but there is no trace of the return response from API in the executor logs.
My_dataframe.repartition(10).foreachPartition(lambda partition: process_partition(partition))
logger.info(temp_dict)
Here I'm storing the response in temp_dict
