Change the log level of Driver:

%scala 
 
spark.sparkContext.setLogLevel("DEBUG")
 
spark.sparkContext.setLogLevel("INFO")

Change the log level of a particular package in Driver logs:

%scala
 
org.apache.log4j.Logger.getLogger("shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.services").setLevel(org.apache.log4j.Level.TRACE)

Change the log level in the executor

sc.parallelize(Seq("")).foreachPartition(x => {
  import org.apache.log4j.{LogManager, Level}
  import org.apache.commons.logging.LogFactory
 
  LogManager.getRootLogger().setLevel(Level.DEBUG)
  val log = LogFactory.getLog("EXECUTOR-LOG:")
  log.debug("START EXECUTOR DEBUG LOG LEVEL")
})

https://kb.databricks.com/clusters/set-executor-log-level.html