Custom docker container for GPU compute using python 3.12

[knocheeri]

Hello!

I have a GPU compute using my own custom docker container. I am trying to upgrade from python 3.10 to 3.12 since 3.10 EOL is next year, but I cannot find any official documentation around this for Databricks runtime.

My current working solution uses databricksruntime/gpu-base:cuda11.8 (based on this official example: https://github.com/databricks/containers/blob/master/ubuntu/gpu/cuda-11.8/venv/Dockerfile) which supports python 3.10, but unfortunately it does not natively support python 3.12. I installed python 3.12 from source still using databricksruntime/gpu-base:cuda11.8, but can't seem to get it to work when running in Databricks with runtime 16.4 LTS and g4dn.xlarge [T4]. 

Has anyone come up with workaround or an alternative solution to this? 

[Louis_Frolio · Accepted solution]

Greetings @knocheeri ,
After doing some research, it looks like there is currently no official support for Python 3.12 (classic compute clusters) in custom GPU containers. At the moment, the highest officially supported version on GPU runtimes is Python 3.10.  To be clear, I am referring to classic clusters where you are allowed to install libraries, not serverless.

The GitHub example you referenced only provides native support for Python 3.10. While I’ve come across anecdotal reports of people attempting to force Python 3.12 into GPU containers, these efforts typically fail due to incompatibilities between driver/worker processes, Python path mismatches, and broader runtime environment conflicts.

Additionally, Databricks has not published any documented or officially tested upgrade path for moving GPU custom containers to Python 3.12. This means that even if you managed to build a custom Docker image with Python 3.12, you’d likely hit instability issues when integrating with the Databricks runtime (CUDA drivers, Spark executors, ML libraries, and other tightly coupled dependencies).

I hope this provides more context around the limitations you’re running into.

Cheers, Louis.

[knocheeri]

Yeah, I have been down the rabbit hole of trying to force 3.12 into GPU containers to no avail. With 3.10 end of life next year, I hope that Databricks is working on an official solution.