How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect?

[Olaoye_Somide]

Hello Databricks Community,

I am currently working in a Databricks environment and trying to set up custom logging using Log4j in a Python notebook. However, I've run into a problem due to the use of Spark Connect, which does not support the _jvm attribute necessary for accessing Log4j directly.

Here's the code I'm using:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
log4j = spark._jvm.org.apache.log4j
logger = log4j.LogManager.getLogger("com.databricks.example")
logger.setLevel(log4j.Level.ERROR)
logger.error("This is a test error message.")

Upon executing, I receive the following error:

[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `_jvm` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark Connect when creating your session.

Could anyone suggest an alternative approach for implementing custom logging in a Python notebook on Databricks that complies with the restrictions of Spark Connect? Are there Python-native methods or libraries that integrate well with Databricks for this purpose?

Thank you in advance for your assistance!

[arpit]

import logging

logging.getLogger().setLevel(logging.WARN)

log = logging.getLogger("DATABRICKS-LOGGER")

log.warning("Hello")