I currently have multiple jobs (each running its own job cluster) for my spark structured streaming pipelines that are long running 24x7x365 on DBR 9.x/10.x LTS. My SLAs are 24x7x365 with 1 minute latency.

I have already accomplished the following cost saving opportunities:

1. Using job cluster instead of general purpose compute
2. The trigger interval is processing at 1 minute interval
3. Using fair-share scheduler pools
4. Tuned the worker VM SKU type based on utilization

Given the above, are the following additional cost saving configurations proven* to meet the above streaming SLAs and supported** by Databricks?

1. Spot instances
2. Auto-scaling
3. The motivation for exploring these 2 cost saving options is because streaming data has different message volume (high and low) during different time of the day.
4. Any new additional cost saving options not mentioned so far are also welcome.

* Proven == empirical results in some large scale production scenario for some extended period of time to prove its robustness.

** Supported == Stateful streaming and recoveries supported by the current Spark 3.x APIs

For context, I have already applied the current (2022-04-14) best practices written by Databricks.

Any references for and against "Spot instances" and "Auto-scaling" are appreciated.

Thank you!