topic Issues when trying to modify log4j.properties in Data Engineering

Issues when trying to modify log4j.properties

digui — Fri, 15 Jul 2022 20:58:57 GMT

Hi y'all.

I'm trying to export metrics and logs to AWS cloudwatch, but while following their tutorial to do so, I ended up facing this error when trying to initialize my cluster with an init script they provided.

This is the part where the script fail

# configure cloudwatch agent for driver & executor
if  [  ! -z $DB_IS_DRIVER ] && [ $DB_IS_DRIVER = TRUE ] ; then
  sed -i '/^log4j.appender.publicFile.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties
  sed -i '/log4j.appender.publicFile=com.databricks.logging.RedactionRollingFileAppender/a log4j.appender.publicFile.layout=com.databricks.labs.log.appenders.JsonLayout' /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties
else
   sed -i '/^log4j.appender.console.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties
   sed -i '/log4j.appender.console.layout=org.apache.log4j.PatternLayout/a log4j.appender.console.layout=com.databricks.labs.log.appenders.JsonLayout' /home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties
fi

This is the error:

sed: can't read /home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties: No such file or directory

I'm not familiar with java at all, so It's being kinda hard to understand what I should do here. Should I install log4j manually or something?

Thanks in advance!

Diego

Re: Issues when trying to modify log4j.properties

Vidula — Mon, 05 Sep 2022 08:12:50 GMT

Hi @Diego Rodrigues

Does @Kaniz Fatma response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Re: Issues when trying to modify log4j.properties

atulec016 — Fri, 26 May 2023 11:17:50 GMT

I am facing the same issue . error details

|+ tree /home/ubuntu/databricks/spark/dbconf | |+ cat /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j2.xml | |+ sed -i '/^log4j.appender.publicFile.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties | |sed: can't read /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties: No such file or directory

Any lead would be appreciated

Re: Issues when trying to modify log4j.properties

cool_cool_cool — Mon, 06 May 2024 15:40:25 GMT

@digui

Did you figure out what to do? We're facing the same issue, the script works for the executors.
I was thinking on adding an if that checks if there is log4j.properties and modify it only if it exists