Can I use OSS Spark History Server to view the EventLogs

brickster_2018
Databricks Employee
Databricks Employee

Is it possible to run the OSS SPark history server and view the spark event logs.

brickster_2018
Databricks Employee
Databricks Employee

Yes, it's possible. The OSS Spark history server can read the Spark event logs generated on a Databricks cluster.

Using Cluster log delivery, the SPark logs can be written to any arbitrary location. Event logs can be copied from there to the storage directory pointed by the OSS Spark History server.

View solution in original post