Unexpected Script Execution Differences on databricks.com vs Mobile-Triggered Runtimes

[EllieFarrell]

I’m noticing some unusual inconsistencies in how scripts execute on databricks.com compared to when the same workflow is triggered through a mobile-based API. On Databricks, the script runs perfectly when executed directly inside a cluster notebook. But when triggered through an API call, the execution timing and even some output behaviors change slightly — which ends up affecting downstream tasks. To check whether this is a Databricks-specific behavior or a broader runtime/environment issue, I started comparing execution across other platforms as well. I even tested how lightweight mobile script executors handle similar runtime variations (for example, tools like Delta Executor APK ). Surprisingly, the pattern of environment-dependent execution differences appears in multiple platforms. So my question to the community is: What typically causes scripts to behave differently between direct cluster execution on databricks.com and API-triggered runs? Could it be: Environment variables? Session initialization? Cluster warm-up? API gateway timeout differences? Or something else affecting runtime consistency? Any insights will help a lot — trying to determine if this is a Databricks-side factor or a universal runtime behavior issue.

[bianca_unifeye · Accepted solution]

Hi Ellie,

What you’re seeing is actually quite common , the same script can behave slightly differently when:

• run interactively in a notebook on a cluster, vs
• run as a job / via API trigger (or from a mobile wrapper hitting that API).

It’s usually not “Databricks being random”, but a mix of  different environments and lifecycle. A few typical causes:

1. Different cluster types & configs

In many setups:

Notebook runs → on an all-purpose (interactive) cluster

 API / job runs → on a job cluster or a different pool

Those clusters can differ in:

• Runtime version (DBR), Spark / Scala / Python versions
• Node type / size, autoscaling configs
• Spark configs (shuffle, partitions, broadcast thresholds, timeouts, etc.)
• Installed libraries / init scripts

Even small config differences can change timing and sometimes behaviour (e.g. shuffles, joins, broadcast vs shuffle strategy, timeouts, etc.).

Check:  compare the cluster JSON or Spark UI → Environment for both runs.

1. 2. Stateful notebook vs stateless job

Notebook sessions are stateful:

• You may have cached tables, temp views, broadcast variables already loaded
• Python / Scala variables defined in earlier cells
• Spark configs changed during experimentation
• Data cached in memory or on local disk

When you trigger via API, the job run usually starts with a clean, fresh context:

• No prior caches, temp views, or globals
• No “warm” JVM / Python runtime, everything has to spin up from scratch

That alone can easily explain different timings and sometimes corner-case behaviour if the script accidentally relies on prior state.

1. 3. Identity, permissions & environment variables

API-triggered runs often use:

 A service principal or technical user

 Potentially different default catalog / schema workspace, or permissions

 Different secret scopes / environment variables(set in job config vs cluster)

If your script reads from:

• `dbutils.secrets.get(...)`
• `os.environ[...]`
• default database / catalog (without fully qualifying paths)

…those differences can influence output or error paths.

1. 4. Cluster lifecycle & “warm-up”

Job clusters or on-demand clusters go through:

• Cold start: spin up nodes, start Spark, load libraries
• JIT warm-up: JVM + Python processes optimising code paths during execution

The first run (especially via API) can be noticeably slower than:

• a long-running interactive cluster that’s already “hot”
• a notebook where you’ve already run some heavy cells

If you’re measuring timing precisely, cold vs warm states will show up.

1. API gateway / orchestration differences

When you go through: a mobile app → API gateway → Databricks Jobs API

you also introduce:

• HTTP timeouts / retries
• Slightly different error handling
• Extra latency before the job even starts

This doesn’t usually change results, but it does change timings, and if downstream systems have strict time budgets or expect logs in a specific order, you may notice differences.

How I’d debug / stabilise this

1. Log environment at the start of the script(for both notebook & API runs):

• `spark.version`, DBR runtime
• `spark.conf.getAll` (or at least key configs)
• `os.environ` subset (env vars your script uses)
• current user / service principal (`spark.sql("SELECT current_user()")`)

1. Make the code stateless & parameterised:

• Don’t rely on notebook globals or earlier cells
• Don’t rely on “whatever catalog/schema I happen to be in” – fully qualify tables & paths

1. Use the same cluster config for both paths:

• Either run the notebook on the same job cluster
• Or configure the job to use the same all-purpose cluster, just for testing

1. Warm-up if necessary:

   For very time-sensitive workflows, consider a small “warm-up” call before the real workload (or keep a cluster/pool warm).

 Is this Databricks-specific or universal?

What you observed with other platforms and mobile script executors is spot on:

this is largely a universal “environment + lifecycle” behaviour, not unique to Databricks.

Databricks just makes the differences more visible because:

• interactive clusters are long-lived and stateful
• job / API runs are short-lived and stateless by design

If you can share a minimal example (e.g. same script, cluster configs, and rough timing logs), the community can help narrow down exactly which of the above is biting you most.

Hope this helps clarify what to look at!

[EllieFarrell · Accepted solution]

Thanks for the detailed breakdown — this actually helps a lot.
Your point about stateful vs stateless execution makes complete sense. I also realized that part of my confusion came from comparing runtimes across very different environments.

While investigating “environment-dependent execution differences,” I was testing a few non-Databricks platforms as reference points too — including a lightweight mobile script executor (deltaexecutorkey.com) — and interestingly, the same cold/warm start and context differences show up there as well.

Not related to Databricks directly, of course, but it helped me understand that the behavior I’m seeing isn’t unique to Spark or DBR — it’s more about how each runtime initializes and manages state.

I’ll gather the environment configs (DBR version, spark.conf, env vars) from both sides and share a minimal reproducible example soon. Thanks again for the clarity.