topic Re: Fully serverless databricks SaaS in Data Engineering

Fully serverless databricks SaaS

Avinash_Narala — Wed, 13 Nov 2024 13:39:38 GMT

I'm exploring Databricks' fully serverless SaaS option, as shown in the attached image, which promises quick setup and $400 in initial credits.

I'm curious about the pros and cons of using this fully serverless setup.

Specifically, would this option be suitable for large enterprises with extensive data processing and infrastructure needs, or would a PaaS (Platform as a Service) setup be more beneficial in such cases?

Are there any limitations in the serverless model that might impact scalability, control, or performance for a large organization?

Thank you in advance for your insights!

Re: Fully serverless databricks SaaS

gchandra — Wed, 13 Nov 2024 13:41:49 GMT

There are; if you have spark config, customer jars, and init scripts, they won't work. Please check this page for long list of limitations.

https://docs.databricks.com/en/compute/serverless/limitations.html

Re: Fully serverless databricks SaaS

Avinash_Narala — Wed, 13 Nov 2024 13:46:31 GMT

@gchandra thanks for the reply.

these are the limitations for serverless compute in notebooks, but I'm asking about serverless databricks. The entire set-up will be serverless, basically a SaaS databricks. Now what we are using is PaaS databricks.