topic Re: Way of using pymc.model_to_graphviz into a Databricks notebook in Machine Learning

Way of using pymc.model_to_graphviz into a Databricks notebook

matte — Tue, 22 Nov 2022 08:49:44 GMT

Hi everybody,

I created a simple bayesian model using the pymc library in Python. I would like to graphically represent my model using the pymc.model_to_graphviz(model=model) method.

However, it seems it does not work within a databrcks notebook, even after having installed the library "graphviz" in the cluster. In particular, the error that I get is the following:

ExecutableNotFound: failed to execute PosixPath('dot'), make sure the Graphviz executables are on your systems' PATH

Is there a way to overcome this problem?

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

jonathan-dufaul — Thu, 24 Nov 2022 20:56:38 GMT

A really basic question...do you have graphviz itself installed? Looks like pymc doesn't require it for installation (see for instance https://docs.pymc.io/en/v3/api/model_graph.html)

%pip install graphviz

does it say that it's already installed? I had a similar issue in cython with using libgsl (I thought the package requiring it had installed it, but no)

edit: if anyone is coming here from google in the future. the solution above in itself isn't complete. pip only installs the python bindings for the graphviz executables (as detailed below). you'll still need to install graphviz either through apt or conda.

edit 2: if anyone is here from google in the future, your frustration in solving this problem is entirely valid and so are you 🙂 ... hopefully this solution works for your problem too.

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

matte — Mon, 28 Nov 2022 10:27:39 GMT

Yes, I have installed graphviz library directly to the cluster I use.

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

jonathan-dufaul — Mon, 28 Nov 2022 13:06:22 GMT

I'm really sorry. It looks like graphviz needs to be installed from apt. I just totally thought the pypi package included the executable, but it looks like it doesn't (someone with a similar problem was here)

I believe `%sh apt install -y graphviz` should make pymc work (only on the driver node, so just for testing). When it comes to installing it to the cluster itself, I usually go to someone in the technology department to have them change whatever installation script for the cluster/startup. I believe it just does the above command but in an init script.

It's really throwing me for a loop that pypi didn't include an executable.

Let me know if this doesn't work. We'll get to the bottom of this.

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

matte — Mon, 28 Nov 2022 13:36:06 GMT

Thank you very much! by using:

%sh apt install -y graphviz

directly on the notebook cell it does work. You've been really helpful.

Now when I run:

pymc.model_to_graphviz(model=model)

I get:

<graphviz.graphs.Digraph at 0x7f7fb2634d90>

I also tried to use the "render" method:

pm.model_to_graphviz(model=hierarchical).render()

But the output is just the string:

'.gv.pdf'

Do you know if it is possible to render the graph inside the notebook? Or otherwise where to find the graph?

Once again, thank you very much

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

jonathan-dufaul — Mon, 28 Nov 2022 15:59:32 GMT

preface that I'm not exactly in love with the solution below. does this work?

graph = pm.model_to_graphviz(model=hierarchical)
displayHTML(graph.pipe(format="svg").decode("utf-8"))

also the reason it gives you that .pdf type thing is because by default it renders it to a pdf, and you set the filename using the filename parameter. you can do different formats (pdf,png,svg,...)

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

matte — Mon, 28 Nov 2022 16:23:16 GMT

Yes it works!

Thank you very much! I tried that solution but without the "format" parameter in the .pipe method and it didn't work.

Thank again for your help.

Re: Way of using pymc.model_to_graphviz into a Databricks notebook

Own — Fri, 02 Dec 2022 12:07:06 GMT

%sh apt install -y graphviz