Hi @aupres,

Do you see any failures in spark logs?

Few things to validate:

It appears that the log files are not being generated in the specified directory due to a misconfiguration in your log4j2.properties fil

Check the Appender Configuration:

Ensure that the appender.file is correctly configured to write logs to the specified directory. The File property should point to the exact file path where you want the logs to be written.

appender.file=org.apache.log4j.RollingFileAppender

appender.file.File=C:/spark-4.0.0-preview2-bin-hadoop3/logs/spark.log

appender.file.MaxFileSize=10MB

appender.file.MaxBackupIndex=10

appender.file.layout=org.apache.log4j.PatternLayout

appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

Ensure Correct Log4j2 Configuration: Verify that the log4j2.properties file is being correctly referenced in your Spark configuration. The paths specified in spark.executor.extraJavaOptions and spark.driver.extraJavaOptions should point to the correct log4j2.properties file.

spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j2.properties

spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///C:/spark-4.0.0-preview2-bin-hadoop3/conf/log4j2.properties

Verify Directory Permissions: Ensure that the directory C:/spark-4.0.0-preview2-bin-hadoop3/logs exists and that the user running the Spark application has write permissions to this directory