It looks like this may be a possibility now? I haven't actually tried it, but I noticed a parameter named "only" has been added to the Databricks SDK for when running a job.

Here is the commit that made the change: [Release] Release v0.38.0 (#826) · databricks/databricks-sdk-py@d516d1e

And here is the line on the main branch as of right now: databricks-sdk-py/databricks/sdk/service/jobs.py at f2472bf98160bd4556efbb575b68c5375b2e902e · datab...

The Databricks API docs doesn't display this parameter, but it seems this is a feature that is either coming or already works. It may be worth trying out.