Databricks job cluster logs
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2024 08:10 PM
Hello, how can I enable Databricks to generate a separate spark log for each job run?
What parameters should I use in spark configuration?
Labels:
- Labels:
-
Spark
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2024 01:10 PM
@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs.
But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a new feature. you can raise it is an idea - https://www.databricks.com/feedback.

