topic Re: Cluster doesn't support Photon with Docker Image enabled in Data Engineering

Cluster doesn't support Photon with Docker Image enabled

Pragan — Thu, 03 Feb 2022 08:44:12 GMT

I enabled Photon 9.1 LTS DBR in cluster that was already using Docker Image of the latest version, when I ran a SQL QUery using my cluster, I could not see any Photon engine working in my executor that should be actually running in Photon Engine.

When I disable docker image from my cluster, I could see Photon engine running. Whether I have to enable some feature to support Photon in my docker image or does docker does not Photon feature?

Re: Cluster doesn't support Photon with Docker Image enabled

Anonymous — Thu, 03 Feb 2022 16:55:38 GMT

Hello, @Praganessh S! Welcome to the community. 🙂

My name is Piper and I'm a moderator for the community. Let's give it a while to see what your peers have to say. If we need to, we'll swing back around.

Thank you for your patience.

Re: Cluster doesn't support Photon with Docker Image enabled

-werners- — Fri, 04 Feb 2022 08:10:09 GMT

Interesting...

The fact that it does not use docker makes me think the photon functionality is not (yet) copied to the docker images.

Maybe someone from Databricks can confirm.

Re: Cluster doesn't support Photon with Docker Image enabled

Anonymous — Tue, 29 Mar 2022 16:37:58 GMT

Hello @Praganessh S , Photon is currently in Public Preview. The only way to use it is to explicitly run Databricks-provide Runtime images which contain it. Please see: https://docs.databricks.com/runtime/photon.html#databricks-clusters

and

https://docs.databricks.com/clusters/configure.html#photon-images