Hi Kris, thank you for your answer!

The main reason we are leaning toward serverless is cost efficiency during idle periods. Our workload is very spiky, for most of the day traffic is low, but during short (unpredictable) peak windows we can receive many events at the same time. For example, we might need 40 workers during a peak hour, but we don’t want to pay for 40 idle workers during the rest of the day

I understand this is not the recommended approach, but I'm curious if this may work: a readstream were the log stream is filtered for its own id:

query = ( spark.readStream .format("delta") .table("control_log") .filter(col("target_run_id") == run_id) .writeStream .foreachBatch(handle_tasks) .start()

Would this be an approach?