Serverless Custom Environment Imaging

[AlexM]

Hi,

I'm looking at moving from job clusters to serverless environments. Ideally to reduce cost and improve start up time.
I can see that it is now possible to specify a custom environment .yaml file - and specify Python packages to be installed.

Is there any mechanism to 'pre-install' or use a container image for the serverless? As these packages can take a while to install. I read that there is some caching going on - but the details are a bit opaque?

Would appreciate any advice,

Thanks,
Alex

 

[Ashwin_DSA · Accepted solution]

Hi @AlexM

There isn’t currently a way to bring a pre-built container image into serverless notebooks/jobs. Serverless supports custom environment YAML files and dependency installation/caching, but Databricks Container Services isn’t supported on serverless compute.

So if the goal is to reduce startup time and avoid repeated installs, the best-supported path today is usually to use a workspace-based environment... which is a reusable YAML spec that defines the serverless environment version plus your additional Python packages. Those base environments are pre-built and cached, helping notebooks and jobs start faster.

Also note that Init scripts / compute policies aren’t available on serverless, so environment customisation needs to go through the Environment panel / YAML route rather than cluster bootstrap logic. Databricks says it automatically caches the notebook virtual environment, so reopening an existing notebook usually doesn’t require reinstalling everything again, even after inactivity. That cache behaviour also helps jobs when tasks in the same run share the same dependency set.

For serverless jobs using custom base environments, only the dependencies required for the task are installed at runtime. If your use case needs fully baked images or OS/system-level packages, I’d probably look at standard or dedicated compute with Databricks Container Services instead of serverless.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.