Thanks for the detailed breakdown — this actually helps a lot.
Your point about stateful vs stateless execution makes complete sense. I also realized that part of my confusion came from comparing runtimes across very different environments.

While investigating “environment-dependent execution differences,” I was testing a few non-Databricks platforms as reference points too — including a lightweight mobile script executor (deltaexecutorkey.com) — and interestingly, the same cold/warm start and context differences show up there as well.

Not related to Databricks directly, of course, but it helped me understand that the behavior I’m seeing isn’t unique to Spark or DBR — it’s more about how each runtime initializes and manages state.

I’ll gather the environment configs (DBR version, spark.conf, env vars) from both sides and share a minimal reproducible example soon. Thanks again for the clarity.