Databricks JAR task type functionality
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 02:15 AM
Hi, I would like to understand Databricks JAR based workflow tasks. Can I interpret JAR based runs to be something like a spark-submit on a cluster? In the logs, I was expecting to see the
spark-submit --class com.xyz --num-executors 4
etc., And, then there is another task type introduced recently I think — spark submit task type. If we have a JAR to execute, does spark submit task type and JAR type are implicitly working similar or different?
- Labels:
-
Task
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2023 02:17 PM
Hi, Could you please go through https://docs.databricks.com/workflows/jobs/how-to-use-jars-in-workflows.html and let us know if this helps?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2023 04:25 AM
Hi, I did try using the Workflows>Jobs>CreateTask>JarTaskType>UploadedMyJAR and Class and created JobCluster and tested this task. This JAR reads some tables as input, does some transformations and output as writing some other tables. I would like to know that this JAR task type can be interpreted as
spark-submit --class com.xyz --num-executors job? Because I was searching for this sort of entry in the log file , but couldnt find any. How does the JAR task type works in principle, is my actual question?
