Databricks Community

mrstevegross

I am trying to run a job with (1) custom containers, and (2) via an instance pool. Here's the setup:The custom container is just the DBR-provided `databricksruntime/standard:12.2-LTS`The instance pool is defined via the UI (see screenshot, below).At ...

mrstevegross

We'd like to adopt Databricks instance pool in order to reduce instance-acquisition times (a significant contributor to our test latency). Based on my understanding of the docs, the main levers we can control are: min instance count, max instance cou...

mrstevegross

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...

mrstevegross

I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error: ImportError...

mrstevegross · 02-13-2025

When launching a job via "Create and trigger a one-time run" (docs), when using a custom image (docs), what's the lifetime of the container? Does it create the cluster, start the container, run the job, then terminate the container? Or does the runni...

mrstevegross

Awesome, we're always using init scripts. So we should measure the delta between "creating" and "init_scripts_started". Cool, I'll do that!

mrstevegross

If the goal is to pre-warm 100 instances in the Databricks Instance Pool, you could create a temporary job that will request instances from the pool. This ensures that Databricks provisions the required instances before the actual test run.The cluste...

mrstevegross

>If all 100 instances are needed at once (fully parallel execution):>• Pre-warm the pool: Around 1h 55m before each test run, trigger 100 API calls to acquire instances ahead of time.>•Set idle_termination = 0 to ensure instances shut down as soon as...

mrstevegross

I also see there is a "list cluster events" API (https://docs.databricks.com/api/workspace/clusters/events); can I get the event log this way?

mrstevegross

Followup: I see that there is documentation on exporting Spark logs (https://docs.databricks.com/aws/en/compute/clusters-manage#event-log) and init script logging (https://docs.databricks.com/aws/en/init-scripts/logs#init-script-events). I've been ab...

Databricks Community

User Stats

User Activity

Attempt to use a custom container with an instance pool fails

Possible to programmatically adjust Databricks instance pool more intelligently?

How best to measure the time-spent-waiting-for-an-instance?

How to resolve "cannot import name 'Iterable' from 'collections'" error?

Container lifetime?

Re: How best to measure the time-spent-waiting-for-an-instance?

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

Re: Is it possible to obtain a job's event log via the REST API?

Re: Is it possible to obtain a job's event log via the REST API?