Databricks spark_jar_task failed when submitted via API

Nisha2 — Fri, 23 Feb 2024 05:25:20 GMT

Hello,
We are submitting jobs to the data bricks cluster using /api/2.0/jobs/create this API and running a spark java application (jar that is submitted to this API). We are noticing Java application is executing as expected. however, we see that the status of the job in Databricks is shown as failed at the end.

Can you please help us in resolving this?

We are getting following error in the log:

24/01/24 08:36:01 INFO SparkContext: Successfully stopped SparkContext 24/01/24 08:36:01 INFO ProgressReporter$: Removed result fetcher for 645810760151386822_6024284017790236785_job-734546952940362-run-453202441019294-action-3199680969853150 24/01/24 08:36:01 WARN ScalaDriverWrapper: Spark is detected to be down after running a command 24/01/24 08:36:01 WARN ScalaDriverWrapper: Fatal exception (spark down) in ReplId-8f661-76f6f-2cac6 com.databricks.backend.common.rpc.SparkStoppedException: Spark down: at com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:651) at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:744) at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:520) at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:436) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:279) at java.lang.Thread.run(Thread.java:750) 24/01/24 08:36:03 INFO DrainingState: Started draining: min wait 10000, grace period 5000, max wait 15000. 24/01/24 08:36:05 WARN DriverDaemon: Unexpected exception: java.lang.NullPointerException java.lang.NullPointerException

Also attaching the Snapshot.

Thank you.

Re: Databricks spark_jar_task failed when submitted via API

Nisha2 — Mon, 18 Mar 2024 09:52:39 GMT

Hello @Retired_mod,
Thank you for the reply, after analyzing my code. I got to know that I was trying to create SparkSession inside the try block which automatically terminates the session when the process is finished. So for now I have removed try block. But now I'm facing another issue. Providing the error log below:

24/03/18 09:41:55 ERROR MicroBatchExecution: Query [id = 88d62783-1d5d-4a7e-b75c-875fa5d42892, runId = 24ec7106-189a-49f8-88ec-b4cb213d1496] terminated with error com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Incompatible format detected. You are trying to write to `abfss://pdpdeltalake@stpdpdeltalakepoc.dfs.core.windows.net/test/di/sap-cic-plant-plant/` using Delta, but there is no transaction log present. Check the upstream job to make sure that it is writing using format("delta") and that you are trying to write to the table base path.

topic Databricks spark_jar_task failed when submitted via API in Data Engineering

Databricks spark_jar_task failed when submitted via API

Re: Databricks spark_jar_task failed when submitted via API