cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Is anyone knows how to use python logger in Databricks python job on serverless

mo_moattar
New Contributor III

I'm trying to use the standard Python logging framework in the Databricks jobs instead of print. I'm doing this by using 

spark._jvm.org.apache.log4j.LogManager.getLogger(__name__)

however as I'm running this on serverless, I get the following error 

[JVM_ATTRIBUTE_NOT_SUPPORTED] Directly accessing the underlying Spark driver JVM using the attribute '_jvm' is not supported on serverless compute. If you require direct access to these fields, consider using a single-user cluster. For more details on compatibility and limitations, check: https://learn.microsoft.com/azure/databricks/release-notes/serverless.html#limitations
---------------------------------------------------------------------------
2 REPLIES 2

menotron
Valued Contributor

Yes, direct access to JVM is not allowed on Spark Connect or serverless compute. You could use the python logging framework to log into the output stream handle or any other Handler though.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
stream_handler.setFormatter(formatter)

logger.addHandler(stream_handler)

 

mo_moattar
New Contributor III

I did the same thing however the logs don't show up on the execution of the tasks so I took a

import logging


class LoggerBuilder:
    def __init__(self, log_level: int = logging.INFO) -> None:
        self.logger = logging.getLogger()
        self.logger.setLevel(log_level)
        formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
        console_handler = PrintHandler()
        console_handler.setLevel(log_level)
        console_handler.setFormatter(formatter)
        self.logger.addHandler(console_handler)

    def get_logger(self):
        return self.logger


class PrintHandler(logging.Handler):
    def __init__(self) -> None:
        logging.Handler.__init__(self=self)

    def emit(self, record: logging.LogRecord) -> None:
        print(f"{record.levelname}: {record.getMessage()}")

different approach with a similar structure.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group