Databricks Community

mrstevegross

We'd like to adopt Databricks instance pool in order to reduce instance-acquisition times (a significant contributor to our test latency). Based on my understanding of the docs, the main levers we can control are: min instance count, max instance cou...

mrstevegross

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...

mrstevegross

I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error: ImportError...

mrstevegross · 02-13-2025

When launching a job via "Create and trigger a one-time run" (docs), when using a custom image (docs), what's the lifetime of the container? Does it create the cluster, start the container, run the job, then terminate the container? Or does the runni...

mrstevegross · 01-31-2025

I'm analyzing the performance of a DBR/Spark request. In this case, the cluster is created using a custom image, and then we run a job on it.I've dived into the "Spark UI" part of the DBR interface, and identified 3 jobs that appear to account for an...

mrstevegross

Awesome, we're always using init scripts. So we should measure the delta between "creating" and "init_scripts_started". Cool, I'll do that!

mrstevegross

If the goal is to pre-warm 100 instances in the Databricks Instance Pool, you could create a temporary job that will request instances from the pool. This ensures that Databricks provisions the required instances before the actual test run.The cluste...

mrstevegross

>If all 100 instances are needed at once (fully parallel execution):>• Pre-warm the pool: Around 1h 55m before each test run, trigger 100 API calls to acquire instances ahead of time.>•Set idle_termination = 0 to ensure instances shut down as soon as...

mrstevegross

I also see there is a "list cluster events" API (https://docs.databricks.com/api/workspace/clusters/events); can I get the event log this way?

mrstevegross

Followup: I see that there is documentation on exporting Spark logs (https://docs.databricks.com/aws/en/compute/clusters-manage#event-log) and init script logging (https://docs.databricks.com/aws/en/init-scripts/logs#init-script-events). I've been ab...

Databricks Community

User Stats

User Activity

Possible to programmatically adjust Databricks instance pool more intelligently?

How best to measure the time-spent-waiting-for-an-instance?

How to resolve "cannot import name 'Iterable' from 'collections'" error?

Container lifetime?

How to identify the goal of a specific Spark job?

Re: How best to measure the time-spent-waiting-for-an-instance?

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

Re: Possible to programmatically adjust Databricks instance pool more intelligently?

Re: Is it possible to obtain a job's event log via the REST API?

Re: Is it possible to obtain a job's event log via the REST API?