How to setup Instance profile for initializing Databricks Cluster using Docker?

[Juniper_AIML]

1. I was trying to start of the Databricks cluster through a docker image. I followed the setup instruction. Excluding the additional setup to setup the IAM role and instance profile as I was facing issues.

The image is stored on AWS ECR in a public repo and can be pulled up easily. I have NOT been able to setup the IAM role or the instance profile as mentioned. As soon as I click on  settings -> manage account it logs me out of databricks into the login screen and the same process repeats. (Attached image for the same.

I have been trying to just paste the docker container URL in the cluster instance settings as in the image below without any instance profile.               

 

[Anonymous]

Hello again, @Aman Gaurav​. Thanks for your second question. As before, we'll see what the community has to say first. 🙂

[Kaniz · Accepted solution]

Hi @Aman Gaurav​ ,

Please check the below requirements to avail the Databricks Container Services.

Note :-

Databricks Runtime for Machine Learning and Databricks Runtime for Genomics does not support Databricks Container Services.

• Databricks Runtime 6.1 or above. If you have previously used Databricks Container Services you must upgrade your base images. See the latest images in https://github.com/databricks/containers tagged with 6.x.

• Your Databricks workspace must have Databricks Container Services enabled.

• Your machine must be running a recent Docker daemon (one that is tested and works with Client/Server Version 18.03.0-ce) and the docker command must be available on your PATH.

Source