https://community.databricks.com/t5/data-engineering/how-to-get-node-and-executor-id-and-log/m-p/93398#M38695 <P>any update on this???</P> Thu, 10 Oct 2024 07:12:02 GMT mayur_05 2024-10-10T07:12:02Z https://community.databricks.com/t5/data-engineering/how-to-get-node-and-executor-id-and-log/m-p/91447#M38165 <P>Hi Team,</P><P>We have df has 70 M row count and we are calling an API for 6000 rows per set using df.repartition(rep_count).foreachpartition(func_name) so in func_name we are calling API post req for that partition but when we are trying to print/log something in that partition<STRONG> we can't see that we can see only in executor's stderr log</STRONG> so how we can save this log(volume) in realtime and store it in table and how we can find the id of executor and node so logging purpose&nbsp;</P> Mon, 23 Sep 2024 13:47:04 GMT https://community.databricks.com/t5/data-engineering/how-to-get-node-and-executor-id-and-log/m-p/91447#M38165 mayur_05 2024-09-23T13:47:04Z https://community.databricks.com/t5/data-engineering/how-to-get-node-and-executor-id-and-log/m-p/93398#M38695 <P>any update on this???</P> Thu, 10 Oct 2024 07:12:02 GMT https://community.databricks.com/t5/data-engineering/how-to-get-node-and-executor-id-and-log/m-p/93398#M38695 mayur_05 2024-10-10T07:12:02Z