Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2024 07:11 AM
You will need to leverage cluster-level Spark configurations or global init scripts. This will allow you to set "spark.scheduler.poo" property automatically for all workloads on the cluster.
You can try navigationg to "Compute", select the cluster you want to modify, click edit then look under "Advanced Options." Expand the "Spark" tab and add "spark.scheduler.pool=<your pool name>".
You can verify your setting by running the following command in a notebook that is attached to your cluster:
spark.sparkContext.setLocalProperty("spark.scheduler.pool", "<your pool name>"
Cheers, Louis.