Spark config ignored in job run settings
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-25-2026 07:19 AM
i am talking about this setting:
i tried so far:
spark.executor.cores 8
and
spark.log.level INFO
Both documented here: https://spark.apache.org/docs/latest/configuration.html
but neither i see effect nor i see them set if i check sparkui -> environment tab. there the settings are not set at all.
here the details of an actual run:
Am i doing it wrong? Or is it reset/ ignored for some other reason?
So how to set specific spark settings - in environment, NOT in code?
Labels:
- Labels:
-
Partner