Databricks Community

mrstevegross

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible v...

mrstevegross

I'm running a DBR/Spark job using a container. I've set docker_image.url to `docker.io/databricksruntime/standard:13.3-LTS`, as well as the Spark env var `DATABRICKS_RUNTIME_VERSION=13.3`. At runtime, however, I'm encountering this error: ImportError...

mrstevegross

When launching a job via "Create and trigger a one-time run" (docs), when using a custom image (docs), what's the lifetime of the container? Does it create the cluster, start the container, run the job, then terminate the container? Or does the runni...

mrstevegross · 01-31-2025

I'm analyzing the performance of a DBR/Spark request. In this case, the cluster is created using a custom image, and then we run a job on it.I've dived into the "Spark UI" part of the DBR interface, and identified 3 jobs that appear to account for an...

mrstevegross · 01-23-2025

I'm using a customer container *and* init scripts. At runtime, I get this error:Cluster '...' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:i-0440ddd3a2d5cce79, databricks_error_message:Cluster scoped init script...

mrstevegross

I also see there is a "list cluster events" API (https://docs.databricks.com/api/workspace/clusters/events); can I get the event log this way?

mrstevegross

Followup: I see that there is documentation on exporting Spark logs (https://docs.databricks.com/aws/en/compute/clusters-manage#event-log) and init script logging (https://docs.databricks.com/aws/en/init-scripts/logs#init-script-events). I've been ab...

mrstevegross · 01-27-2025

Followup: I got the AWS creds working by amending our AWS role to permit read/write access to our S3 bucket. Woohoo!

mrstevegross · 01-27-2025

Thanks, that's pretty much what I did; a lot of terraform configuration to get the AWS account set up properly, and now I'm able to tell DBR to load the container. (FWIW, I'm encountering *new* access issues; I started a thread here (https://communit...

mrstevegross · 01-27-2025

>If you are using, for example, an all-purpose cluster with shared access mode, I recommend configuring the "Init Script" option inside the advanced cluster settings.Yep, that's the approach. I've got init scripts specified in the cluster settings, a...

Databricks Community

User Stats

User Activity

How best to measure the time-spent-waiting-for-an-instance?

How to resolve "cannot import name 'Iterable' from 'collections'" error?

Container lifetime?

How to identify the goal of a specific Spark job?

How to grant custom container AWS credentials for reading init script?

Re: Is it possible to obtain a job's event log via the REST API?

Re: Is it possible to obtain a job's event log via the REST API?

Re: How to grant custom container AWS credentials for reading init script?

Re: Format when specifying docker_image url?

Re: How to grant custom container AWS credentials for reading init script?