Databricks - autostart from jdbc query

nickmaco — Wed, 21 Jul 2021 08:26:27 GMT

Hi team,

New to Databricks and trying to understand if there is a "True" auto-start capability with Databricks. We are evaluating Databricks Delta lake as an alternative cloud based datawarehouse solution but the biggest problem I see is the inability to allow a cluster to auto-start.

Even setting up a pool of idle VMs slightly improves the start up time for clusters of different sizes but there doesn't seem to be a way to auto-start a cluster upon a jdbc request from an external client. (IE business user making a BI request)

What I have seen in the community is references to start up scripts which are scheduled which would be fine for known workloads but I'm referring to a scenario where the usage is unknown. So needs to be automated.

Let me know if there is something obvious I'm missing but can't seem to see a solution with Databricks which means it will sit back in the Datascience/ML environment and leaving other Cloud Based Datawarehouses to run the BI workloads.

Re: Databricks - autostart from jdbc query

nickmaco — Wed, 21 Jul 2021 12:53:02 GMT

Just adding on to this.

Using DBeaver as a client and using a singlenode cluster and a pool of idling VM, it was possible to get the autostart time of the cluster down to 35 seconds, + 17 seconds for the query time on top to show the first 200 rows of a 500,000 record object doesn't really compare to the other Datawarehouse SaaS products out there.

Even looked to make sure I was not using ML runtime.

Welcome others thoughts and opinions on what approach to take or just leave as not viable.

topic Databricks - autostart from jdbc query in Data Engineering

Databricks - autostart from jdbc query

Re: Databricks - autostart from jdbc query