cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Enabling serverless type for SQL warehouse running on Google Cloud Platform

96286
Contributor

I am in the process of connecting Looker to one of my Databricks databases. To reduce startup time on my SQL warehouse cluster I would like to change the type from "Pro" to "Serverless". I cannot find a way to do that and "Serverless" is not an option for me. 

Is Serverless not available when running databricks on GCP? I can't find much information in the docs... Thanks. 

4 REPLIES 4

Kaniz_Fatma
Community Manager
Community Manager

Hi @96286 , As of now, serverless SQL warehouses are exclusive to Azure Databricks and are not available on Databricks running on other cloud platforms, like GCP.

If you're using Databricks on GCP and want to enhance the startup time of your SQL warehouse cluster, you can consider the following strategies:

  1. Cache Frequently Accessed Tables: Use the .cache() or .persist() method to cache frequently accessed tables. This will expedite query processing by having the data readily available in memory.

  2. Cluster Initialization Scripts: Leverage Databricks' cluster initialization scripts. These scripts help automate the installation of essential libraries and dependencies, ensuring your cluster is pre-configured with the necessary components.

  3. Ephemeral Clusters: Explore the newly launched feature called "ephemeral clusters" in Databricks on GCP. Ephemeral clusters are lightweight and short-lived, optimized for rapid startup times, typically launching within seconds. They can help reduce the overall cost of running Databricks on GCP.

While serverless SQL endpoints may not be accessible on Databricks on GCP at this time, these optimization strategies can help you streamline your cluster startup times and manage costs effectively.

glawry
New Contributor II

Are Ephemeral Clusters still a feature that is offered? I do not see any docs on them, and I don't see them as an option in the SQL Warehouse Create UI.  

Kayla
Contributor III

Echoing glawry - I'd be fascinated to know if this "Ephemeral clusters" are a thing.

Well currently I believe ephemeral clusters are offered on Google Cloud (GCP) Dataproc (tin boxes). Dataproc is mainly for running Spark and Hadoop jobs on GCP (you can add other artefacts to Dataproc as well like Flink etc) . Dataproc is optimized for launching ephemeral clusters quickly for short-lived jobs. So serverless is not on offer on GCP. However, you can utilize autoscaling clusters to adjust size based on workload. On GCP, BigQuery is serverless. 

Mich Talebzadeh | Technologist | Data | Generative AI | Financial Fraud
London
United Kingdom

view my Linkedin profile



https://en.everybodywiki.com/Mich_Talebzadeh



Disclaimer: The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner Von Braun)".
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!