Hey, your approach is a common pattern. But there has been a recent shift where we now recommend creating the agents as databricks apps instead. This allows you to not do pipeline execution steps. Here are a couple of articles you may find helpful.

https://www.databricks.com/blog/custom-agents-now-available-databricks

https://docs.databricks.com/aws/en/generative-ai/agent-framework/migrate-agent-to-apps

Let me know if you need anything else.

Thanks,

Emma