cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ckwan48
by New Contributor III
  • 1594 Views
  • 4 replies
  • 1 kudos

Create a Dockerfile from Cluster

Is there a way to create a Dockerfile from Workspace A's cluster configurations and deploy that on a different different cluster in Workspace B?

  • 1594 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Kevin Kim​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...

  • 1 kudos
3 More Replies
tibfab
by New Contributor II
  • 2646 Views
  • 5 replies
  • 0 kudos

How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?

I successfully built a custom docker image for the Standard runtime following the steps described on the page Customize containers with Databricks Container Services and based on the image databricksruntime/standard:11.3-LTS. However, I cannot find ...

  • 2646 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Tibor Fabian​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

  • 0 kudos
4 More Replies
nachog99
by New Contributor II
  • 3323 Views
  • 5 replies
  • 2 kudos

Databricks cluster starts with docker

Hi there!I hope u are doing wellI'm trying to start a cluster with a docker image to install all the libraries that I have to use.I have the following Dockerfile to install only python libraries as you can seeFROM databricksruntime/standard WORKDIR /...

image.png
  • 3323 Views
  • 5 replies
  • 2 kudos
Latest Reply
xneg
Contributor
  • 2 kudos

Hi! I am facing a similar issue.I tried to use this oneFROM databricksruntime/standard:10.4-LTS   ENV DEBIAN_FRONTEND=noninteractive RUN apt update && apt install -y maven && rm -rf /var/lib/apt/lists/*   RUN /databricks/python3/bin/pip install datab...

  • 2 kudos
4 More Replies
SoniSole
by New Contributor II
  • 3474 Views
  • 5 replies
  • 6 kudos

Issue with Docker Image connection

Hello, I have created and pushed a docker image to Azure Container Registry . I used that image to start the cluster in Databricks. But the cluster doesn't start and as such when I try to run a Databricks' Jon using that Cluster, I get this error bel...

image
  • 3474 Views
  • 5 replies
  • 6 kudos
Latest Reply
dsfrederic
New Contributor II
  • 6 kudos

  • 6 kudos
4 More Replies
umarkhan
by New Contributor II
  • 2119 Views
  • 4 replies
  • 1 kudos

Driver context not found for python spark for spark_submit_task using Jobs API submit run endpoint

I am trying to run a multi file python job in databricks without using notebooks. I have tried setting this up by:creating a docker image using the DBRT 10.4 LTS as a base and adding the zipped python application to that.make a call to the run submit...

  • 2119 Views
  • 4 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Umar Khan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
3 More Replies
askme
by New Contributor II
  • 1040 Views
  • 2 replies
  • 2 kudos

Databricks jobs UPdate/Reset API throws an unexpected error

{ "error_code": "INVALID_PARAMETER_VALUE", "message": "Missing required field: job_id"}I have a test job cluster and I need to update the docker image filed with the other version using reset/update job API. I went through the documentation of data b...

  • 1040 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hey there @radha kilaru​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 2 kudos
1 More Replies
ivanychev
by Contributor
  • 983 Views
  • 3 replies
  • 3 kudos

AWS + Cluster with custom docker image = export metrics to CloudWatch?

Is there a working setup on setting up metrics export to CloudWatch while using custom docker images for cluster creation? I've tried to set up the CloudWatch agent manually, but launching `amzon-cloudwatch-agent-ctl` in the bootstrap script fails wi...

  • 983 Views
  • 3 replies
  • 3 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 3 kudos

We do not support gangila with custom docker too. but let me cross verify if we are supporting cloudwatch for the same. Sorry for the inconvenience @Sergey Ivanychev​ 

  • 3 kudos
2 More Replies
michaelh
by New Contributor III
  • 3090 Views
  • 3 replies
  • 2 kudos

Resolved! Databricks runtime from docker hub image

Hello,are databricks runtimes from docker hub ( https://hub.docker.com/r/databricksruntime/standard ) same as actual runtimes inside Databricks? I mean when we made our own docker image from databricksruntime/standard will be there same dependencies...

image.png image
  • 3090 Views
  • 3 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Hi @michael henzl​ ,Just checking if you still need help with this or not any more? please let us know

  • 2 kudos
2 More Replies
Pragan
by New Contributor
  • 2177 Views
  • 5 replies
  • 1 kudos

Resolved! Cluster doesn't support Photon with Docker Image enabled

I enabled Photon 9.1 LTS DBR in cluster that was already using Docker Image of the latest version, when I ran a SQL QUery using my cluster, I could not see any Photon engine working in my executor that should be actually running in Photon Engine.When...

  • 2177 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hello @Praganessh S​ , Photon is currently in Public Preview. The only way to use it is to explicitly run Databricks-provide Runtime images which contain it. Please see: https://docs.databricks.com/runtime/photon.html#databricks-clustersandhttps://do...

  • 1 kudos
4 More Replies
Rb29
by New Contributor
  • 466 Views
  • 0 replies
  • 0 kudos

Image Display in Dockerized Cluster

I am using a docker recipe for configuring my databricks cluster. It is working fine for everything else however when I tried to display any image data using any python utility such as matplotlib, PIL or Opencv etc. the image does not get displayed o...

  • 466 Views
  • 0 replies
  • 0 kudos
francescocamuss
by New Contributor III
  • 2498 Views
  • 5 replies
  • 0 kudos

Resolved! How to make terminal available when starting a cluster with a docker image?

Hello everybody, I'm starting a cluster with a docker image because I my team use a lot of R libraries that would be time consuming to install them on a init script. The thing is that the cluster starts ok, but I can't acces to the cluster terminal a...

image.png image
  • 2498 Views
  • 5 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Hi @Francesco Camussoni​ , This is one of the limitations of DBR.Limitations of DBR:- Azure Databricks does not support running Spark jobs from the web terminal. In addition, the Azure Databricks web terminal is not available in the following cluster...

  • 0 kudos
4 More Replies
Juniper_AIML
by New Contributor
  • 1716 Views
  • 2 replies
  • 1 kudos

Resolved! How to setup Instance profile for initializing Databricks Cluster using Docker?

I was trying to start of the Databricks cluster through a docker image. I followed the setup instruction. Excluding the additional setup to setup the IAM role and instance profile as I was facing issues.The image is stored on AWS ECR in a public repo...

Screenshot 2022-02-15 at 2.39.57 PM Screenshot 2022-02-15 at 2.49.03 PM
  • 1716 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Aman Gaurav​ , Please check the below requirements to avail the Databricks Container Services.Note :-Databricks Runtime for Machine Learning and Databricks Runtime for Genomics does not support Databricks Container Services.Databricks Runtime 6.1...

  • 1 kudos
1 More Replies
francescocamuss
by New Contributor III
  • 6934 Views
  • 13 replies
  • 10 kudos

Resolved! Databricks rbase container: Rstudio doesn´t work

Hello, How are you? I hope you are doing well!I´m trying to use a databrick´s image (link: containers/ubuntu/R at master · databricks/containers (github.com)) to run a container when starting a cluster. I need that Rstudio is installed on the contain...

1 2 3 5
  • 6934 Views
  • 13 replies
  • 10 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 10 kudos

If the issue is resolved would you be happy to mark the answer as best so that others can quickly find the solution in the future.

  • 10 kudos
12 More Replies
Anonymous
by Not applicable
  • 1266 Views
  • 1 replies
  • 0 kudos

Resolved! Questions on using Docker image with Databricks Container Service

Specifically, we have in mind:* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)* When we want to test an API change, build a Docker image with the relevant changes in a Jar file* Update the job configur...

  • 1266 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

>Where do we put custom Jar files when building the Docker image? /databricks/jars>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to tak...

  • 0 kudos
Labels