Hi @mnissen1337 ,

I would use serverless for that use case. It takes a time for job cluster to spin up (of course you can use pools, but given that your job needs to run every 5 minutes it doesn't make much sense), so serverless seems to be a great fit.

Regarding your concern about installing packages every time. Serveless compute let's you define custom environment that can be reused. What's great  is that environments cache installed packages, which reduces startup latency for subsequent runs. 

If my answer was helpful, please consider marking it as accepted solution.

Configure the serverless environment | Databricks on AWS