Databricks job cluster logs

Jackson1111
New Contributor III

Hello, how can I enable Databricks to generate a separate spark log for each job run?

What parameters should I use in spark configuration?

 

shan_chandra
Databricks Employee
Databricks Employee

@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs. 

But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a new feature. you can raise it is an idea - https://www.databricks.com/feedback