How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect?

Olaoye_Somide — Mon, 22 Apr 2024 06:56:42 GMT

Hello Databricks Community,

I am currently working in a Databricks environment and trying to set up custom logging using Log4j in a Python notebook. However, I've run into a problem due to the use of Spark Connect, which does not support the _jvm attribute necessary for accessing Log4j directly.

Here's the code I'm using:

from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() log4j = spark._jvm.org.apache.log4j logger = log4j.LogManager.getLogger("com.databricks.example") logger.setLevel(log4j.Level.ERROR) logger.error("This is a test error message.")

Upon executing, I receive the following error:

[JVM_ATTRIBUTE_NOT_SUPPORTED] Attribute `_jvm` is not supported in Spark Connect as it depends on the JVM. If you need to use this attribute, do not use Spark Connect when creating your session.

Could anyone suggest an alternative approach for implementing custom logging in a Python notebook on Databricks that complies with the restrictions of Spark Connect? Are there Python-native methods or libraries that integrate well with Databricks for this purpose?

Thank you in advance for your assistance!

Re: How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect?

arpit — Tue, 23 Apr 2024 18:21:06 GMT

import logging

logging.getLogger().setLevel(logging.WARN)

log = logging.getLogger("DATABRICKS-LOGGER")

log.warning("Hello")

topic Re: How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect? in Data Engineering

How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect?

Re: How to Implement Custom Logging in Databricks without Using _jvm Attribute with Spark Connect?