Databricks Community

mrstevegross

Currently, DBR does not permit a user to run a containerized job on a graviton machines (per these docs). In our case, we're running containerized jobs on a pool. We are exploring adopting Graviton, but--per those docs--DBR won't let us do that.Are t...

mrstevegross

I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`The instance pool is defined via the UI (see screenshot, below).At ...

mrstevegross

We'd like to adopt Databricks instance pool in order to reduce instance-acquisition times (a significant contributor to our test latency). Based on my understanding of the docs, the main levers we can control are: min instance count, max instance cou...

mrstevegross

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...

mrstevegross · 02-24-2025

I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error: ImportError...

mrstevegross

I think I have solved this. I added a URL for `preloaded_docker_image` to my instance pool, and the job worked correctly.This suggests that the DBR docs for preloaded_docker_image are incomplete; they should clarify that a user must add an entry in o...

mrstevegross

Did y'all ever figure this out? I'm running in a similar issue.

mrstevegross

>To preload container services in a pool, you must do it via the Databricks API, since this option is not available through the UI.I'm not trying to "preload" it, I just want to my cluster to (1) use a container, and (2) use my pool. I'm aware that t...

mrstevegross

Awesome, we're always using init scripts. So we should measure the delta between "creating" and "init_scripts_started". Cool, I'll do that!

mrstevegross

If the goal is to pre-warm 100 instances in the Databricks Instance Pool, you could create a temporary job that will request instances from the pool. This ensures that Databricks provisions the required instances before the actual test run.The cluste...

Databricks Community

User Stats

User Activity

Graviton & containers?

Attempt to use a custom container with an instance pool fails

Possible to programmatically adjust Databricks instance pool more intelligently?

How best to measure the time-spent-waiting-for-an-instance?

How to resolve "cannot import name 'Iterable' from 'collections'" error?

Re: Attempt to use a custom container with an instance pool fails

Re: Custom docker image fails to initalize

Re: Attempt to use a custom container with an instance pool fails

Re: How best to measure the time-spent-waiting-for-an-instance?

Re: Possible to programmatically adjust Databricks instance pool more intelligently?