iyashk-DB
Databricks Employee
Databricks Employee

Hi @bsr , there was some internal discussion on this going on and I got to know that these DEBUG thread-dump lines from the ThreadMonitor started leaking to stderr/job output due to a Python logger misconfiguration introduced in the 17.3.3 branch. This was not an intentional behavior change, and they rolled out fix which will be released in the next maintenance release. Until then you can use the following approach:

To use a Python snippet to change the Python ThreadMonitor log level.

import logging
logger = logging.getLogger("ThreadMonitor")

for handler in list(logger.handlers):
    if handler.name == "console":
       logger.handlers.remove(handler)

logger.propagate = False

 

View solution in original post