how to generate log files on specific folders
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2024 04:23 PM
Hello! My environments are like below,
OS : Windows 11
Spark : spark-4.0.0-preview2-bin-hadoop3
And the configuration of spark files 'spark-defaults.conf' and 'log4j2.properties'
spark-defaults.conf
spark.eventLog.enabled true
spark.eventLog.dir file:///C:/spark-4.0.0-preview2-bin-hadoop3/sparkeventlogs
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 5g
spark.yarn.am.memory 1g
spark.executor.instances 1
spark.executor.extraJavaOptions -Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j.properties
spark.driver.extraJavaOptions -Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j.properties
log4j2.properties
rootLogger = INFO, console, file
rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console, file
rootLogger.appenderRef.file.ref = FileAppender
appender.file=org.apache.log4j.RollingFileAppender
appender.file.File=C:/spark-4.0.0-preview2-bin-hadoop3/logs
appender.file.MaxFileSize=10MB
appender.file.MaxBackupIndex=10
appender.file.layout=org.apache.log4j.PatternLayout
appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
But log files are not generated on C:/spark-4.0.0-preview2-bin-hadoop3/logs folder. Any idea?