Hello, 

What I'm trying to do : 
I have a piece of python code (an ai application) that i want to deploy as a server, almost exactly like one would lets say on an EC2. The only difference being instead of a flask api i would use dbutils and the databricks API to create a job with my code as a task and then hit the notebook on a pre attached cluster. 

The challenge I'm facing : 
unlike a traditional VM which would add maybe a few seconds of overhead tops, this is adding a huge overhead maybe 30 seconds or more. The actual code needs only 2 seconds but the job finishes execution in about 32 seconds or so. I'm fairly new to databricks and spark in general, so I wanted to know what is causing this underneath the hood. Also if this approach is a bad idea, could someone explain why exactly, and is there a better way of doing this on databricks ?