why doesn't databricks allow setting executor metrics

marc88 — Sat, 01 Feb 2025 19:19:22 GMT

I have an all-purpose compute cluster that processes different data sets for various jobs. I am struggling to optimize executor metrics like below.
spark.executor.memory 4g

Is it allowed to override default executor metrics and specify such configurations at the cluster level for an all-purpose compute cluster? (in Spark config section under Advance cluster options)
How do I specify such configurations at runtime while submitting a job to a job-compute cluster?

Re: why doesn't databricks allow setting executor metrics

Alberto_Umana — Sun, 02 Feb 2025 02:27:33 GMT

Hello @marc88,

As you mentioned in Spark config under Advance cluster options you can do it 🙂 once cluster boots up it will be set at run level. Or you can draft a cluster policy and apply it across for job computes when creating your workflow.

topic why doesn't databricks allow setting executor metrics in Administration & Architecture

why doesn't databricks allow setting executor metrics

Re: why doesn't databricks allow setting executor metrics