Hi Team,
We have df has 70 M row count and we are calling an API for 6000 rows per set using df.repartition(rep_count).foreachpartition(func_name) so in func_name we are calling API post req for that partition but when we are trying to print/log something in that partition we can't see that we can see only in executor's stderr log so how we can save this log(volume) in realtime and store it in table and how we can find the id of executor and node so logging purpose
