Spark executor logs path

[Jhaprakash6608]

We are running spark workloads and have enabled cluster log discovery to push executor logs to Azure blog. While that's running fine, I'd also like to know the local path of the executor logs so that I can make use of oneagent from dynatrace and send it there. 

Appreciate your comments and info.

[Krishna_S]

Local Executor Log Path on Azure Databricks Executor logs are written locally on each executor node under the work directory: The path pattern is:

/databricks/spark/work/<app-id>/<executor-id>

For example: /databricks/spark/work/app-20221121180310-0000/0

This directory contains logs specific to each executor, including standard output (stdout), standard error (stderr), and log4j log files. 

This directory is present on the local filesystem of each Spark worker (executor) node, regardless of whether cluster log delivery (to Azure Blob/DBFS) is enabled.

The log delivery mechanism periodically uploads from these local files—but the original logs reside on the worker node while the executor is running.