cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

vladcrisan
by New Contributor II
  • 3787 Views
  • 5 replies
  • 1 kudos

Can Spark History server be created in Databricks?

We have a Spark pipeline producing more than 3k Spark jobs. After the pipeline finishes and the cluster shuts down, only a subset (<1k) of these can be recovered from the Spark UI.We would like to have access to the full Spark UI after the pipeline t...

  • 3787 Views
  • 5 replies
  • 1 kudos
Latest Reply
Sandeep
Contributor III
  • 1 kudos

@Vlad Crisan​ , you can use the Databricks clusters to replay the events. Please follow this kb: https://kb.databricks.com/clusters/replay-cluster-spark-eventsNote: Please spin up a cluster with version 10.4 LTS.

  • 1 kudos
4 More Replies
Labels