Is anyone knows how to use python logger in Databricks python job on serverless

[mo_moattar]

I'm trying to use the standard Python logging framework in the Databricks jobs instead of print. I'm doing this by using 

spark._jvm.org.apache.log4j.LogManager.getLogger(__name__)

however as I'm running this on serverless, I get the following error 

[JVM_ATTRIBUTE_NOT_SUPPORTED] Directly accessing the underlying Spark driver JVM using the attribute '_jvm' is not supported on serverless compute. If you require direct access to these fields, consider using a single-user cluster. For more details on compatibility and limitations, check: https://learn.microsoft.com/azure/databricks/release-notes/serverless.html#limitations
---------------------------------------------------------------------------

[menotron]

Yes, direct access to JVM is not allowed on Spark Connect or serverless compute. You could use the python logging framework to log into the output stream handle or any other Handler though.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
stream_handler.setFormatter(formatter)

logger.addHandler(stream_handler)

[mo_moattar]

I did the same thing however the logs don't show up on the execution of the tasks so I took a

import logging


class LoggerBuilder:
    def __init__(self, log_level: int = logging.INFO) -> None:
        self.logger = logging.getLogger()
        self.logger.setLevel(log_level)
        formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
        console_handler = PrintHandler()
        console_handler.setLevel(log_level)
        console_handler.setFormatter(formatter)
        self.logger.addHandler(console_handler)

    def get_logger(self):
        return self.logger


class PrintHandler(logging.Handler):
    def __init__(self) -> None:
        logging.Handler.__init__(self=self)

    def emit(self, record: logging.LogRecord) -> None:
        print(f"{record.levelname}: {record.getMessage()}")

different approach with a similar structure.