Databricks Community

mrstevegross

Currently, DBR does not permit a user to run a containerized job on a graviton machines (per these docs). In our case, we're running containerized jobs on a pool. We are exploring adopting Graviton, but--per those docs--DBR won't let us do that.Are t...

mrstevegross

I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`The instance pool is defined via the UI (see screenshot, below).At ...

mrstevegross · 03-17-2025

We'd like to adopt Databricks instance pool in order to reduce instance-acquisition times (a significant contributor to our test latency). Based on my understanding of the docs, the main levers we can control are: min instance count, max instance cou...

mrstevegross · 03-12-2025

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...

mrstevegross · 02-24-2025

I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error: ImportError...

mrstevegross

Ok, good to know  . If anyone from DBR is monitoring this thread, can you confirm lsi's understanding of the product roadmap?

mrstevegross

If anyone from DBR is monitoring this thread, can y'all confirm my understanding and--if so--update the docs to reflect this requirement?

mrstevegross

I think I have solved this. I added a URL for `preloaded_docker_image` to my instance pool, and the job worked correctly.This suggests that the DBR docs for preloaded_docker_image are incomplete; they should clarify that a user must add an entry in o...

mrstevegross

Did y'all ever figure this out? I'm running in a similar issue.

mrstevegross

>To preload container services in a pool, you must do it via the Databricks API, since this option is not available through the UI.I'm not trying to "preload" it, I just want to my cluster to (1) use a container, and (2) use my pool. I'm aware that t...

Databricks Community

User Stats

User Activity

Graviton & containers?

Attempt to use a custom container with an instance pool fails

Possible to programmatically adjust Databricks instance pool more intelligently?

How best to measure the time-spent-waiting-for-an-instance?

How to resolve "cannot import name 'Iterable' from 'collections'" error?

Re: Graviton & containers?

Re: Attempt to use a custom container with an instance pool fails

Re: Attempt to use a custom container with an instance pool fails

Re: Custom docker image fails to initalize

Re: Attempt to use a custom container with an instance pool fails