Issues when trying to modify log4j.properties

[digui]

Hi y'all.

I'm trying to export metrics and logs to AWS cloudwatch, but while following their tutorial to do so, I ended up facing this error when trying to initialize my cluster with an init script they provided.

This is the part where the script fail

# configure cloudwatch agent for driver & executor
if  [  ! -z $DB_IS_DRIVER ] && [ $DB_IS_DRIVER = TRUE ] ; then
  sed -i '/^log4j.appender.publicFile.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties
  sed -i '/log4j.appender.publicFile=com.databricks.logging.RedactionRollingFileAppender/a log4j.appender.publicFile.layout=com.databricks.labs.log.appenders.JsonLayout' /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties
else
   sed -i '/^log4j.appender.console.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties
   sed -i '/log4j.appender.console.layout=org.apache.log4j.PatternLayout/a log4j.appender.console.layout=com.databricks.labs.log.appenders.JsonLayout' /home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties
fi

This is the error:

sed: can't read /home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties: No such file or directory

I'm not familiar with java at all, so It's being kinda hard to understand what I should do here. Should I install log4j manually or something?

Thanks in advance!

Diego

[Kaniz_Fatma]

Hi @Diego Rodrigues​, This is a fantastic KB from Databricks on how to overwrite log4j configurations on Databricks clusters. Please look at it and let us know if that helps.

[Vidula]

Hi @Diego Rodrigues​ 

Does @Kaniz Fatma​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

[atulec016]

I am facing the same issue . error details

|+ tree /home/ubuntu/databricks/spark/dbconf | |+ cat /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j2.xml | |+ sed -i '/^log4j.appender.publicFile.layout/ s/^/#/g' /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties | |sed: can't read /home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties: No such file or directory

Any lead would be appreciated

[cool_cool_cool]

@digui 

Did you figure out what to do? We're facing the same issue, the script works for the executors.
I was thinking on adding an if that checks if there is log4j.properties and modify it only if it exists