Hi @Debayan Mukherjee​ docker image is created using an argo workflow. I used this dockerfile as reference: https://github.com/databricks/containers/blob/master/ubuntu/R/Dockerfile. I'm not sure I follow you 2nd question. Cluster is getting created fine, it is that they are missing all the R packages which should get installed on them bc of dockerfile.

Here's my dockerfile code:

FROM databricksruntime/standard:10.4-LTS

# Suppress interactive configuration prompts

ENV DEBIAN_FRONTEND=noninteractive

ENV DOWNLOAD_STATIC_LIBV8=1

ENV TZ=America/New_York

# install dependencies

RUN apt-get update \

  && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 \

  && add-apt-repository -y 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/' \  

  && apt-get install build-essential --yes \

   dirmngr gnupg apt-transport-https ca-certificates software-properties-common \

   autoconf \

   automake \

   g++ \

   gcc \

   cmake \

   gfortran \

   make \

   nano \

   liblapack-dev \

   liblapack3 \

   libopenblas-base \

   libopenblas-dev \

   libcurl4-openssl-dev\

   libxml2-dev\

   libssl-dev\

   libnlopt-dev \

   r-base \

   r-base-dev \

  && apt-get clean all \

  && rm -rf /var/lib/apt/lists/*

RUN R -e "install.packages(c('remotes', 'shiny'), repos='https://cran.microsoft.com/')" 

#RUN R -e "remotes::install_github('facebookexperimental/Robyn/R');"

RUN R -e "install.packages('Robyn')"

RUN R -e "library(Robyn)"

# # DBI/ODBC dependencies

RUN R -e "install.packages(c('DBI', 'dplyr','dbplyr','odbc'), repos='https://cran.microsoft.com/')"

# # Databricks dependencies

# # hwriterPlus is used by Databricks to display output in notebook cells

# # Rserve allows Spark to communicate with a local R process to run R code

RUN R -e "install.packages(c('hwriterPlus'), repos='https://mran.revolutionanalytics.com/snapshot/2017-02-26')"

RUN R -e "install.packages(c('htmltools'), repos='https://cran.microsoft.com/')"

RUN R -e "install.packages('Rserve', repos='http://rforge.net/')"

RUN R -e "install.packages('reticulate');"

RUN R -e "library(reticulate)"

# ## Install Nevergrad

# # RUN R -e "reticulate::use_python('/opt/conda/bin/python3')"

# # RUN R -e "reticulate::py_config()"

# # RUN R -e "reticulate::py_install('nevergrad', pip = TRUE)"

RUN /databricks/python3/bin/pip install nevergrad