Enabling serverless type for SQL warehouse running on Google Cloud Platform

[96286]

I am in the process of connecting Looker to one of my Databricks databases. To reduce startup time on my SQL warehouse cluster I would like to change the type from "Pro" to "Serverless". I cannot find a way to do that and "Serverless" is not an option for me. 

Is Serverless not available when running databricks on GCP? I can't find much information in the docs... Thanks. 

[Kaniz]

Hi @96286 , As of now, serverless SQL warehouses are exclusive to Azure Databricks and are not available on Databricks running on other cloud platforms, like GCP.

If you're using Databricks on GCP and want to enhance the startup time of your SQL warehouse cluster, you can consider the following strategies:

1. Cache Frequently Accessed Tables: Use the .cache() or .persist() method to cache frequently accessed tables. This will expedite query processing by having the data readily available in memory.

2. Cluster Initialization Scripts: Leverage Databricks' cluster initialization scripts. These scripts help automate the installation of essential libraries and dependencies, ensuring your cluster is pre-configured with the necessary components.

3. Ephemeral Clusters: Explore the newly launched feature called "ephemeral clusters" in Databricks on GCP. Ephemeral clusters are lightweight and short-lived, optimized for rapid startup times, typically launching within seconds. They can help reduce the overall cost of running Databricks on GCP.

While serverless SQL endpoints may not be accessible on Databricks on GCP at this time, these optimization strategies can help you streamline your cluster startup times and manage costs effectively.

[glawry]

Are Ephemeral Clusters still a feature that is offered? I do not see any docs on them, and I don't see them as an option in the SQL Warehouse Create UI.  

[Kayla]

Echoing glawry - I'd be fascinated to know if this "Ephemeral clusters" are a thing.

[MichTalebzadeh]

Well currently I believe ephemeral clusters are offered on Google Cloud (GCP) Dataproc (tin boxes). Dataproc is mainly for running Spark and Hadoop jobs on GCP (you can add other artefacts to Dataproc as well like Flink etc) . Dataproc is optimized for launching ephemeral clusters quickly for short-lived jobs. So serverless is not on offer on GCP. However, you can utilize autoscaling clusters to adjust size based on workload. On GCP, BigQuery is serverless.