Cluster doesn't support Photon with Docker Image enabled

[Pragan]

I enabled Photon 9.1 LTS DBR in cluster that was already using Docker Image of the latest version, when I ran a SQL QUery using my cluster, I could not see any Photon engine working in my executor that should be actually running in Photon Engine.

When I disable docker image from my cluster, I could see Photon engine running. Whether I have to enable some feature to support Photon in my docker image or does docker does not Photon feature?

[Anonymous]

Hello, @Praganessh S​! Welcome to the community. 🙂

My name is Piper and I'm a moderator for the community. Let's give it a while to see what your peers have to say. If we need to, we'll swing back around.

Thank you for your patience.

[-werners-]

Interesting...

The fact that it does not use docker makes me think the photon functionality is not (yet) copied to the docker images.

Maybe someone from Databricks can confirm.

[Anonymous · Accepted solution]

Hello @Praganessh S​ , Photon is currently in Public Preview. The only way to use it is to explicitly run Databricks-provide Runtime images which contain it. Please see: https://docs.databricks.com/runtime/photon.html#databricks-clusters

and

https://docs.databricks.com/clusters/configure.html#photon-images