Hi,

@Brahmareddy is right — I’ve encountered the same issue. Even when using a continuous job, I still experience the overhead of compute restarting after each run completes.

As a temporary workaround (until the more cost-effective serverless update is available), I’ve created a main notebook that uses dbutils.notebook.run inside a while loop to handle orchestration. This loop runs continuously but breaks every few hours to force a compute restart. Because it's a single-task notebook set up as a continuous job, it immediately kicks off a new run after exiting.

I’ve also experimented with compute pools, but they seem to introduce a similar level of overhead.

This setup is far from ideal, but it works for now as we await future improvements from Databricks.