topic Re: Are serverless endpoints possible in this Technical Blog post by qianyu? in Generative AI

Are serverless endpoints possible in this Technical Blog post by qianyu?

brandt6264 — Sat, 03 May 2025 17:35:54 GMT

Hi, are serverless endpoints possible for Whisper and Llama in this Technical Blog post by qianyu?

https://community.databricks.com/t5/technical-blog/streamline-customer-call-center-transcripts-analytics-with/ba-p/101689

Thanks!

Re: Are serverless endpoints possible in this Technical Blog post by qianyu?

-werners- — Mon, 05 May 2025 07:12:34 GMT

Model serving is already serverless enabled. You can even set a budget for this (public preview).
https://docs.databricks.com/aws/en/machine-learning/model-serving/manage-serving-endpoints
https://docs.databricks.com/aws/en/machine-learning/model-serving/foundation-model-overview

Or do you mean something else (like using a databricks serverless sql/eng instance)?

Re: Are serverless endpoints possible in this Technical Blog post by qianyu?

brandt6264 — Mon, 05 May 2025 15:24:43 GMT

Thanks for the clarification and links.

Bottom line, what I am trying to avoid is spinning up AWS resources in the background that will incur ongoing charges until I track them down and terminate them (I am a new Databricks customer and still trying to navigate the billing/cost system). The "Serve this model" UI looked suspiciously like it was going to do this but on second look, maybe not.

I am just wanting to confirm my only costs, Databricks or AWS, for Whisper and Llama in this Technical Blog post will be only for the short duration they will be used. Thanks!

Re: Are serverless endpoints possible in this Technical Blog post by qianyu?

-werners- — Tue, 06 May 2025 07:15:16 GMT

https://www.databricks.com/product/pricing/foundation-model-serving

We don´t serve models on databricks, but as far as i can see you pay per input/output tokens (for foundation models).
For classic models:
https://www.databricks.com/product/pricing/model-serving