DBR 17.3.3 introduced unexpected DEBUG logs from ThreadMonitor – how to disable?

[bsr]

After upgrading from DBR 17.3.2 to DBR 17.3.3, we started seeing a flood of DEBUG logs like this in job outputs:

```

DEBUG:ThreadMonitor:Logging python thread stack frames for MainThread and py4j threads: DEBUG:ThreadMonitor:Logging Thread-8 (run) stack frames: File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap

[...]

```

Our logger configuration is:

logging.basicConfig(level=logging.INFO)

logging.getLogger("py4j.clientserver").setLevel(logging.WARNING)

This worked fine before, but now these `ThreadMonitor` DEBUG messages appear regardless.

Questions:

1. Is this an intentional change in DBR 17.3.3?

2. Is there an official way to disable `ThreadMonitor` DEBUG logs without affecting our current python package?

3. Will this be documented or reverted in future DBR releases?

Any insights or official guidance would be appreciated.

Environment: Databricks Runtime 17.3.3 (Python 3.12, PySpark, ipykernel)

[iyashk-DB · Accepted solution]

Hi @bsr , there was some internal discussion on this going on and I got to know that these DEBUG thread-dump lines from the ThreadMonitor started leaking to stderr/job output due to a Python logger misconfiguration introduced in the 17.3.3 branch. This was not an intentional behavior change, and they rolled out fix which will be released in the next maintenance release. Until then you can use the following approach:

To use a Python snippet to change the Python ThreadMonitor log level.

import logging
logger = logging.getLogger("ThreadMonitor")

for handler in list(logger.handlers):
    if handler.name == "console":
       logger.handlers.remove(handler)

logger.propagate = False

[Domme]

... will be released in the next maintenance release.

@iyashk-DB When can this release be expected?

[bsr]

Thanks for the quick response!

[WAHID]

@iyashk-DB

We are currently using DBR version 17.3 LTS, and the issue is still occurring.

Do you know when the fix is expected to be applied? We need this information to decide whether we should wait for the fix or proceed with the workaround you proposed.

 Thank you.