My main question is why you use spark in the first place, but that is another discussion.

If you want to use databricks, you will have to launch a spark job in one way or another on the databricks cluster with spark-submit or by using the jobs api.

However, if you want to use this with your microservices, I guess you need an always-on databricks cluster, which will become expensive (unless you can wait several minutes for a cluster to spin up).

Or perhaps check out a streaming scenario?