Move to 100% Serverless

[dbrx_user]

Hi all,

A few questions about the upcoming transition to 100% serverless, if anyone has any info that would be great!

• When will the move to serverless occur? I understand from 1st July (today) but has anyone seen a roadmap?
• What will the move to serverless mean in practice? i.e. will existing clusters remain and the ability to create new clusters be turned off or something else?
• What will happen to current managed resource groups?
• What about customers on reserved instances?
• For customers without UC enabled workspaces - will they need to transition?

Thanks!

[Kaniz_Fatma]

Hi @dbrx_user, Let me address your questions about the transition to serverless in Databricks:

1. Serverless Compute for Databricks SQL:

   • Databricks SQL now offers a serverless compute option, which is generally available. This feature provides instant compute for BI and SQL workloads directly on the data lake.
   • With serverless compute, users can run queries without waiting for clusters to start up or scale out. It’s ideal for business analysts who need quick access to real-time datasets.
   • You only pay for serverless computing when running reports or queries, which helps lower overall cos....

2. Transition Timeline:

   • The transition to serverless has been rolling out gradually. As of August 3, 2022, it was available in Azure East US 2, Azure East US, and Azure West Europe.
   • Keep an eye out for further updates and availability in other regions.

3. Existing Clusters and Managed Resource Groups:

   • Existing clusters will continue to function, but serverless computing provides an alternative for BI and SQL workloads.
   • Managed resource groups remain unaffected by the transition.

4. Reserved Instances:

   • Customers using reserved instances can still benefit from serverless compute. It complements existing clusters and provides additional flexibility.

5. UC-Enabled Workspaces:

   • For customers without UC-enabled workspaces, transitioning to serverless is optional. However, it’s a powerful way to simplify compute management and reduce costs.

Remember that Databricks periodically releases updates, so stay informed about any changes or enhanc...³. If you have further questions, feel free to ask!

[dbrx_user]

Thanks @Kaniz_Fatma but I was actually referring to their new announcement from the Data and AI Summit that they would be transitioning to 100% serverless across their whole platform. I haven't seen much beyond the initial announcements in terms of what this actually looks like.

[m997al]

Hi.  I also just heard about this announcement for serverless across the entire Databricks platform (what does that mean for Azure Databricks?).  I can see for some of the new AI features, or even for the SQL warehouses (maybe)...but to have all the compute clusters serverless is not desired at all.

I noticed that you said the 100% serverless, even for Databricks SQL, is optional for Unity Catalog workspaces...I wonder if that applies to the broader announcement at the summit?

At any rate, as the original poster says, there has been very little news on this (to google) since the summit, and I would think it would be a very major thing.  My company cannot, even in the medium-term, use serverless compute at all.  Thanks!

[m997al · Accepted solution]

Hi, so our Databricks contact just assured us the following, after we asked about this issue:

Databricks is officially (but won’t be GA in every region till end of July) 100% Serverless OPTIONAL.

We understand many of our customers have begged for 100% serverless for a long time now, but others may want or need to keep the classic compute in place.

Every company can use both and will be able to select either or at all times. There is no plane to sunset the classic compute model and there is no plane for it to be regarded as legacy in any way.

We simply want to give companies options. Having the Serverless option can also be a good extra step for preventing VM outages in your Azure region for example.

Hope this helps explain that specific product announcement.