I recently wrote a KB article on this. Let me know if this helps!

TLDR: Databricks Apps are lightweight, container-based runtimes designed for UI rendering and light orchestration. They do not ship with an Apache Spark driver, executor, or JVM.

Any call that instantiates a SparkSession (or lower-level SparkContext) tries to start the Java gateway and fails, producing the [JAVA_GATEWAY_EXITED] error.

Databricks Apps should delegate compute to an existing Databricks cluster or to Databricks SQL instead of attempting to create Spark locally.