topic Re: print(flush = True) not working in Data Engineering

print(flush = True) not working

tanjil — Mon, 09 Jan 2023 05:50:11 GMT

Hello,

I have the following minimum example working example using multiprocessing:

from multiprocessing import Pool
 
files_list = [('bla', 1, 3, 7), ('spam', 12, 4, 8), ('eggs', 17, 1, 3)]
 
 
def f(t):
  print('Hello from child process', flush = True)
  
  name, a, b, c = t
  return (name, a + b + c)
 
def main():
  pool = Pool(processes=3) # set the processes max number 3  
  result = pool.map(f, files_list)
  pool.close()
  pool.join()
  print('end')
  print(result)
 
if __name__ == "__main__":
  main()

However, I cannot get anything from the child process to print. The aim is use the text for debugging purposes in the future. Any idea why this does not work or if there are any alternative solution to this? Thank you.

Re: print(flush = True) not working

tanjil — Tue, 10 Jan 2023 09:58:47 GMT

No errors are generated. The code executes successfully, but there the print statement for "Hello from child process" does not work.

Re: print(flush = True) not working

Debayan — Mon, 09 Jan 2023 22:27:52 GMT

Hi, Do you receive any error when you say it is not working?

Re: print(flush = True) not working

drourke — Thu, 02 Jan 2025 01:09:02 GMT

did you ever figure this out? is it because of map? I'm having the same problem..

Re: print(flush = True) not working

drourke — Thu, 02 Jan 2025 06:56:56 GMT

nm (discovered queue)