topic Re: How best to measure the time-spent-waiting-for-an-instance? in Get Started Discussions

How best to measure the time-spent-waiting-for-an-instance?

mrstevegross — Wed, 12 Mar 2025 17:33:38 GMT

I'm exploring using an instance pool. Can someone clarify for me which job event log tells me the time-spent-waiting-for-an-instance? I've found 2 candidates:

1. The delta between "waitingForCluster" and "started" on the "run events" log, accessible via "View Run Events" button on the Runs page (e.g.: `https://tecton-dev-shared-databricks.cloud.databricks.com/jobs/12345/runs/12345`):

2. The delta between "CREATING" and "INIT_SCRIPTS_STARTED" on the "run events" tab on the Compute page (e.g.: `https://tecton-dev-shared-databricks.cloud.databricks.com/compute/clusters/12345/events?o=12345`:

Which delta should I use to assess time-spent-waiting-for-an-instance?

Re: How best to measure the time-spent-waiting-for-an-instance?

Isi — Mon, 17 Mar 2025 22:00:45 GMT

Hey @mrstevegross

I think you are reading the “View Run Events” log in reverse order.
I’m attaching an example where I compare the View Run Events (Right) with the Run Events from the Compute Page (Left).

As you can see, both logs contain the same information, but in the correct order:

CREATING → INIT_SCRIPTS_STARTED (20:44:54 → 20:46:58) ≈ 2 min

If your setup doesn’t include init scripts, the time should be slightly lower, as the cluster wouldn’t need to execute those steps.

Hope this helps 🙂

Isi

Re: How best to measure the time-spent-waiting-for-an-instance?

mrstevegross — Tue, 18 Mar 2025 00:55:11 GMT

Awesome, we're always using init scripts. So we should measure the delta between "creating" and "init_scripts_started". Cool, I'll do that!

Re: How best to measure the time-spent-waiting-for-an-instance?

julieAnderson — Tue, 18 Mar 2025 02:07:24 GMT

System Logs or Event Timings

Re: How best to measure the time-spent-waiting-for-an-instance?

Isi — Tue, 18 Mar 2025 07:50:22 GMT

Hey @mrstevegross

Great! Consider mark my answer as Solution to help future users.

Thanks,

Isi