how to generate log files on specific folders

aupres — Sun, 15 Dec 2024 00:23:28 GMT

Hello! My environments are like below,

OS : Windows 11 Spark : spark-4.0.0-preview2-bin-hadoop3

And the configuration of spark files 'spark-defaults.conf' and 'log4j2.properties'

spark-defaults.conf

spark.eventLog.enabled true spark.eventLog.dir file:///C:/spark-4.0.0-preview2-bin-hadoop3/sparkeventlogs spark.serializer org.apache.spark.serializer.KryoSerializer spark.driver.memory 5g spark.yarn.am.memory 1g spark.executor.instances 1 spark.executor.extraJavaOptions -Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j.properties spark.driver.extraJavaOptions -Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j.properties

log4j2.properties

rootLogger = INFO, console, file rootLogger.level = info rootLogger.appenderRef.stdout.ref = console, file rootLogger.appenderRef.file.ref = FileAppender appender.file=org.apache.log4j.RollingFileAppender appender.file.File=C:/spark-4.0.0-preview2-bin-hadoop3/logs appender.file.MaxFileSize=10MB appender.file.MaxBackupIndex=10 appender.file.layout=org.apache.log4j.PatternLayout appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

But log files are not generated on C:/spark-4.0.0-preview2-bin-hadoop3/logs folder. Any idea?

Re: how to generate log files on specific folders

Alberto_Umana — Sun, 15 Dec 2024 00:58:13 GMT

Hi @aupres,

Do you see any failures in spark logs?

Few things to validate:

It appears that the log files are not being generated in the specified directory due to a misconfiguration in your log4j2.properties fil

Check the Appender Configuration:

Ensure that the appender.file is correctly configured to write logs to the specified directory. The File property should point to the exact file path where you want the logs to be written.

appender.file=org.apache.log4j.RollingFileAppender

appender.file.File=C:/spark-4.0.0-preview2-bin-hadoop3/logs/spark.log

appender.file.MaxFileSize=10MB

appender.file.MaxBackupIndex=10

appender.file.layout=org.apache.log4j.PatternLayout

appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

Ensure Correct Log4j2 Configuration: Verify that the log4j2.properties file is being correctly referenced in your Spark configuration. The paths specified in spark.executor.extraJavaOptions and spark.driver.extraJavaOptions should point to the correct log4j2.properties file.

spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j2.properties

spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j2.properties

Verify Directory Permissions: Ensure that the directory C:/spark-4.0.0-preview2-bin-hadoop3/logs exists and that the user running the Spark application has write permissions to this directory

topic how to generate log files on specific folders in Data Engineering

how to generate log files on specific folders

Re: how to generate log files on specific folders