cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

adrianna2942842
by New Contributor III
  • 8736 Views
  • 3 replies
  • 6 kudos

Resolved! Exception "java.nio.charset.MalformedInputException: Input length = 1" when creating data profile on Docker Container Service (10.4 LTS)

I am encountering an issue while attempting to create a data profile on clusters using Docker Container Service (version 10.4 LTS). I keep receiving the following exception:java.nio.charset.MalformedInputException: Input length = 1What's puzzling is ...

  • 8736 Views
  • 3 replies
  • 6 kudos
Latest Reply
Vartika
Moderator
  • 6 kudos

Hi @Adrianna Klank​,We haven't heard from you since the last response from @Akash Bhat​​, and I was checking back to see if the suggestion helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to other...

  • 6 kudos
2 More Replies
ckwan48
by New Contributor III
  • 1651 Views
  • 4 replies
  • 1 kudos

Create a Dockerfile from Cluster

Is there a way to create a Dockerfile from Workspace A's cluster configurations and deploy that on a different different cluster in Workspace B?

  • 1651 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Kevin Kim​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...

  • 1 kudos
3 More Replies
nachog99
by New Contributor II
  • 3447 Views
  • 5 replies
  • 2 kudos

Databricks cluster starts with docker

Hi there!I hope u are doing wellI'm trying to start a cluster with a docker image to install all the libraries that I have to use.I have the following Dockerfile to install only python libraries as you can seeFROM databricksruntime/standard WORKDIR /...

image.png
  • 3447 Views
  • 5 replies
  • 2 kudos
Latest Reply
xneg
Contributor
  • 2 kudos

Hi! I am facing a similar issue.I tried to use this oneFROM databricksruntime/standard:10.4-LTS   ENV DEBIAN_FRONTEND=noninteractive RUN apt update && apt install -y maven && rm -rf /var/lib/apt/lists/*   RUN /databricks/python3/bin/pip install datab...

  • 2 kudos
4 More Replies
229031
by New Contributor II
  • 844 Views
  • 1 replies
  • 1 kudos

Using your own docker container to launch databricks cluster.

When using your own docker container while creating a databricks cluster, what is the mapping between the number of containers launched and the nodes launched?Is it 1:1 mapping? or is it similar to other orchestration framework like Kubernetes?Or is ...

  • 844 Views
  • 1 replies
  • 1 kudos
Latest Reply
FRG96
New Contributor III
  • 1 kudos

+1

  • 1 kudos
Tacuma
by New Contributor II
  • 995 Views
  • 4 replies
  • 1 kudos

Scheduling jobs with Airflow result in each task running multiple jobs.

Hey everyone, I'm experiementing with running containerized pyspark jobs in Databricks, and orchestrating them with airflow. I am however, encountering an issue here. When I trigger an airflow DAG, and I look at the logs, I see that airflow is spinni...

  • 995 Views
  • 4 replies
  • 1 kudos
Latest Reply
Tacuma
New Contributor II
  • 1 kudos

Both, I guess? Yes, all jobs share the same config - the question I have is why in the same airflow task log, there are 3 jobs runs. I'm hoping that there's something in the configs and may give me some kind of clue.

  • 1 kudos
3 More Replies
HQJaTu
by New Contributor III
  • 1208 Views
  • 2 replies
  • 2 kudos

Custom container doesn't launch systemd

Quite soon after moving from VMs to containers, I started crafting my own images. That way notebooks have all the necessary libraries already there and no need to do any Pipping/installing in the notebook.As requirements get more complex, now I'm at ...

  • 1208 Views
  • 2 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi @Jari Turkia​ , Please check if this helps: https://developers.redhat.com/blog/2019/04/24/how-to-run-systemd-in-a-container#other_cool_features_about_podman_and_systemdAlso, you can run ubuntu /redhat linux OS inside containers which will have sys...

  • 2 kudos
1 More Replies
dsura
by Contributor
  • 4254 Views
  • 7 replies
  • 19 kudos

Resolved! Azure AAD token with Databricks for User defined managed Identity inside Docker Container

Hi, We are currently using a Azure AAD Token inorder to authenticate with Databricks instead of generating Personal Access Tokens from Databricks. We have a multi-tenant architecture and so we are using Azure container instances to run multiple trans...

  • 4254 Views
  • 7 replies
  • 19 kudos
Latest Reply
Kaniz
Community Manager
  • 19 kudos

Hi @Dharit Sura​ ​, We haven’t heard from you since the last response from @Debayan Mukherjee​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others....

  • 19 kudos
6 More Replies
LadislavSulak
by New Contributor II
  • 1013 Views
  • 3 replies
  • 2 kudos

What is the long-term plan for the official Databricks Docker Containers?

Hi, I'd like to ask you, how much resources do you plan to dedicate to the maintenance/development of the official Databricks Docker images, please? Do you have a view on the longer-term plan for these docker images? It seems to be maintained, but i...

  • 1013 Views
  • 3 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

Curious too,but I have a feeling it is not a priority.

  • 2 kudos
2 More Replies
williamsmith900
by New Contributor
  • 1454 Views
  • 2 replies
  • 1 kudos

How to remove old and unused Docker images

When running Docker for a long time, there are a lot of images in the system. How can I remove all unused Docker images at once safely to free up the storage?In addition, I also want to remove images pulled months ago, So, I'm not asking for removing...

  • 1454 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @william smith​ Hope everything is going great!Does @Prabakar Ammeappin​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you...

  • 1 kudos
1 More Replies
ivanychev
by Contributor
  • 1033 Views
  • 3 replies
  • 3 kudos

AWS + Cluster with custom docker image = export metrics to CloudWatch?

Is there a working setup on setting up metrics export to CloudWatch while using custom docker images for cluster creation? I've tried to set up the CloudWatch agent manually, but launching `amzon-cloudwatch-agent-ctl` in the bootstrap script fails wi...

  • 1033 Views
  • 3 replies
  • 3 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 3 kudos

We do not support gangila with custom docker too. but let me cross verify if we are supporting cloudwatch for the same. Sorry for the inconvenience @Sergey Ivanychev​ 

  • 3 kudos
2 More Replies
Maverick1
by Valued Contributor II
  • 2072 Views
  • 5 replies
  • 7 kudos

How to deploy mlflow models to sagemaker endpoints where sagemaker refers the private docker registry?

Is it possible to deploy the mlflow model to a sagemaker endpoint where the image URL is not referring to an image in ECR but the image is actually present in a private docker registry?

  • 2072 Views
  • 5 replies
  • 7 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 7 kudos

@Saurabh Verma​ , this to create the endpoint.also, check this out - https://github.com/mlflow/mlflow/blob/0fa849ad75e5733bf76cc14a4455657c5c32f107/mlflow/sagemaker/__init__.py#L361

  • 7 kudos
4 More Replies
michaelh
by New Contributor III
  • 3154 Views
  • 3 replies
  • 2 kudos

Resolved! Databricks runtime from docker hub image

Hello,are databricks runtimes from docker hub ( https://hub.docker.com/r/databricksruntime/standard ) same as actual runtimes inside Databricks? I mean when we made our own docker image from databricksruntime/standard will be there same dependencies...

image.png image
  • 3154 Views
  • 3 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Moderator
  • 2 kudos

Hi @michael henzl​ ,Just checking if you still need help with this or not any more? please let us know

  • 2 kudos
2 More Replies
francescocamuss
by New Contributor III
  • 7141 Views
  • 13 replies
  • 10 kudos

Resolved! Databricks rbase container: Rstudio doesn´t work

Hello, How are you? I hope you are doing well!I´m trying to use a databrick´s image (link: containers/ubuntu/R at master · databricks/containers (github.com)) to run a container when starting a cluster. I need that Rstudio is installed on the contain...

1 2 3 5
  • 7141 Views
  • 13 replies
  • 10 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 10 kudos

If the issue is resolved would you be happy to mark the answer as best so that others can quickly find the solution in the future.

  • 10 kudos
12 More Replies
Anonymous
by Not applicable
  • 1309 Views
  • 1 replies
  • 0 kudos

Resolved! Questions on using Docker image with Databricks Container Service

Specifically, we have in mind:* Create a Databricks job for testing API changes (the API library is built in a custom Jar file)* When we want to test an API change, build a Docker image with the relevant changes in a Jar file* Update the job configur...

  • 1309 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

>Where do we put custom Jar files when building the Docker image? /databricks/jars>How do we update the job configuration so that the job’s cluster will be built with this new Docker image, and how long do we expect this re-configuring process to tak...

  • 0 kudos
Labels