Custom logging using Log4J to a file

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hello,

I would like to ask for help setting up the log4j.

I want to use log4j (log4j2) to generate custom log messages in my notebook when running.This message would be generated like this: logger.info("some info message") but using log4j not python logging module.

I want to store only this custom messages into a file on the driver for example under /tmp folder and the file name should be as name of the notebook.

It can be similar to this solution Writing PySpark logs in Apache Spark and Databricks | by Ivan Trusov | Medium , but it stores everything under same file and also with many other different log messages.

So is there something I can do to reach this logging requirement?

Thank you.