topic Re: Databricks JAR task type functionality in Data Engineering

Databricks JAR task type functionality

Chanu — Wed, 11 Jan 2023 10:15:40 GMT

Hi, I would like to understand Databricks JAR based workflow tasks. Can I interpret JAR based runs to be something like a spark-submit on a cluster? In the logs, I was expecting to see the

spark-submit --class com.xyz --num-executors 4

etc., And, then there is another task type introduced recently I think — spark submit task type. If we have a JAR to execute, does spark submit task type and JAR type are implicitly working similar or different?

Re: Databricks JAR task type functionality

Chanu — Thu, 12 Jan 2023 12:25:52 GMT

Hi, I did try using the Workflows>Jobs>CreateTask>JarTaskType>UploadedMyJAR and Class and created JobCluster and tested this task. This JAR reads some tables as input, does some transformations and output as writing some other tables. I would like to know that this JAR task type can be interpreted as

spark-submit --class com.xyz --num-executors

job? Because I was searching for this sort of entry in the log file , but couldnt find any. How does the JAR task type works in principle, is my actual question?

Re: Databricks JAR task type functionality

Debayan — Wed, 11 Jan 2023 22:17:26 GMT

Hi, Could you please go through https://docs.databricks.com/workflows/jobs/how-to-use-jars-in-workflows.html and let us know if this helps?