topic How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)? in Data Engineering

How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

tibfab — Thu, 09 Feb 2023 15:32:23 GMT

I successfully built a custom docker image for the Standard runtime following the steps described on the page Customize containers with Databricks Container Services and based on the image databricksruntime/standard:11.3-LTS. However, I cannot find an appropriate base image for an ML runtime.

The only thing I can come up with is to export the list of all Python packages via pip freeze from a notebook in a running ML cluster and build the Docker image based on that list, but that seems tedious.

Also I'm not sure even that could work with the base image for the Standard runtime. There might be other dependencies missing apart from Python packages.

Has anyone an idea how this could be achieved?

Re: How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

tibfab — Mon, 13 Feb 2023 19:09:15 GMT

12.1 ML

Re: How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

Anonymous — Fri, 17 Feb 2023 04:57:13 GMT

Hi @Tibor Fabian

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

tibfab — Fri, 17 Feb 2023 07:04:52 GMT

Hi, no I'm not. There is still no Docker image in Docker Hub to customize runtime ML 12.1 easily.

Re: How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

Anonymous — Mon, 10 Apr 2023 07:23:45 GMT

Hi @Tibor Fabian

Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

Re: How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

Debayan — Mon, 13 Feb 2023 16:54:37 GMT

Hi,

Could you please confirm which ML runtime version is needed to create custom docker image?