Spark executor logs path

Jhaprakash6608 — Fri, 13 Jun 2025 04:39:25 GMT

We are running spark workloads and have enabled cluster log discovery to push executor logs to Azure blog. While that's running fine, I'd also like to know the local path of the executor logs so that I can make use of oneagent from dynatrace and send it there.

Appreciate your comments and info.

Re: Spark executor logs path

Krishna_S — Wed, 15 Oct 2025 15:58:39 GMT

Local Executor Log Path on Azure Databricks Executor logs are written locally on each executor node under the work directory: The path pattern is:

/databricks/spark/work/<app-id>/<executor-id>

For example: /databricks/spark/work/app-20221121180310-0000/0

This directory contains logs specific to each executor, including standard output (stdout), standard error (stderr), and log4j log files.

This directory is present on the local filesystem of each Spark worker (executor) node, regardless of whether cluster log delivery (to Azure Blob/DBFS) is enabled.

The log delivery mechanism periodically uploads from these local files—but the original logs reside on the worker node while the executor is running.

topic Spark executor logs path in Administration & Architecture

Spark executor logs path

Re: Spark executor logs path