How do I change the log level in Databricks?

brickster_2018 — Thu, 24 Jun 2021 06:33:44 GMT

How can I change the log level of the Spark Driver and executor process?

Re: How do I change the log level in Databricks?

brickster_2018 — Thu, 24 Jun 2021 06:47:51 GMT

Change the log level of Driver:

%scala 
 
spark.sparkContext.setLogLevel("DEBUG")
 
spark.sparkContext.setLogLevel("INFO")

Change the log level of a particular package in Driver logs:

%scala
 
org.apache.log4j.Logger.getLogger("shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services").setLevel(org.apache.log4j.Level.TRACE)

Change the log level in the executor

sc.parallelize(Seq("")).foreachPartition(x => {
  import org.apache.log4j.{LogManager, Level}
  import org.apache.commons.logging.LogFactory
 
  LogManager.getRootLogger().setLevel(Level.DEBUG)
  val log = LogFactory.getLog("EXECUTOR-LOG:")
  log.debug("START EXECUTOR DEBUG LOG LEVEL")
})

https://kb.databricks.com/clusters/set-executor-log-level.html

topic How do I change the log level in Databricks? in Data Engineering

How do I change the log level in Databricks?

Re: How do I change the log level in Databricks?