@Atanu Sarkar​  @Gobinath Viswanathan​ @Kaniz Fatma​ :

I have been trying to push a registered model in DB managed mlflow to sagemaker endpoint. Although I have been able to do it but there are some manual steps that I needed to do on my local system in order to make it work. Could you help me to understand, Am I doing it correctly or Is there a bug in the Databricks ML runtime.

Below are the steps that I did:

1. Step1: Log the model 
   1. Ran a model code and registered it in mlflow. Moved the model into production stage.
2. Step2: Deploy the model
   1. Installed AWS CLI (via pip) and configured the AWS target env./account. This account have a role ARN setup with Sagemaker full access and ECRContainerRegistry full access.
   2. Able to connect the target AWS account via databricks notebook.
   3. While I was deploying the model as Sagemaker endpoint via “mlflow.sagemaker.deploy”, All intermediate obejcts are being created but I was getting the error because it was not able to find the container image in ECR. My initial assumption was that the function itself should be able to create containers by using the current model code.

So, I  downloaded the model files into a folder on DB local path using mlflow library.

1. Now In order to create a container, I am using “mlflow sagemaker build-and-push-container” command from the DB local path where model files are present.It is showing me error “no module named docker” from the “mlflow.docker_utils” module.
2. In order to resolve this I did “pip install docker”. But after that I am getting the error: docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"

I have checked that this error comes when the docker daemon processes itself are not working. I also haven’t been able to find any docker process executable file in “/etc/init.d/” path where general service executables are present.

The only way all things works is when I downloaded all model based files on my local system, ran the docker desktop for docker daemons to be up and then ran “mlflow sagemaker build-and-push-container” command from inside the models folder. It had created an image in the ECR which is being correctly referred by “mlflow.sagemaker.deploy” command.

My question is that, Is this the right process? Do we need to build the image locally in order to make it work?  

My assumption was that the “mlflow.sagemaker.deploy” command would be able to take care of all things Or atmost the “mlflow sagemaker build-and-push-container” command should be able to run from databricks notebook itself.