Hi!

So I used this MLFlow experiment I found from the databricks website: https://docs.databricks.com/_static/notebooks/machine-learning-with-unity-catalog.html

And I created this cluster using a custom Docker image I created myself:  

Usually when I create a cluster to run an MLFlow experiment, I use the ML version such as 10.4 LTS ML (includes Apache Spark 3.2.1, Scala 2.12). However, when I tried to select this for the Docker image, this error shows up:

So I thought, I'll just use the regular 10.4 LTS one. But when I used this and ran my MLFlow experiment, nothing shows up in the experiment runs.

Why is this happening? What can I do to make this work?

Please help! Thank you very much!