Databricks

igorgatis · ‎10-11-2023

I find it quite hard to understand Spark UI for my pyspark pipelines. For example, when one writes `spark.read.table("sometable").show()` it shows:

I learned that `DataFrame` API actually may spawn jobs before running the actual job. In the example above, job 15 collects data which is used in job 16. In both cases, the description gives no clue on what is going on.

Clicking on job 15 link, it shows a stage that looks like this:

Whose link leads to:

The job 16 is quite similar though it mentions the table name. Things get messier when DAG gets more complex.

Is there a recommended way to improve this? I'm aware of `setJobDescription`, `setLocalProperty` (with `callSite.short` and `callSite.long` but dealing with them directly is also not easy.

Kaniz · ‎10-16-2023

Hi @igorgatis ,

The Spark User Interface can be complex, especially for new users who are not familiar with the framework. However, there are a few tips and techniques that can make the UI more understandable.

Use meaningful job and stage names. When you submit a job to Spark, you can set the job and stage names to be more descriptive of what each job or stage is doing. This can help you identify jobs more easily in the UI and make it easier to track their progressThe Spark UI can be a bit overwhelming, especially for large or complex pipelines. Fortunately, there are a few ways to improve the visibility and understandability of your Spark jobs.
Use meaningful job, stage, and task names: You can use the setJobDescription, setName, and setName methods in PySpark or Scala to provide descriptive names for each job, stage, and task in your pipeline. This can help you quickly identify which part of the pipeline is being executed and what it is doing.
Enable query logging: Spark supports query logging, which logs all executed SQL queries along with their associatedUnderstanding the Spark UI and tracing the data flow can be intimidating, especially for complex or long-running Spark jobs and queries. Here are a few strategies you can use to improve your experience with the Spark UI:
Monitoring Spark Jobs The Spark UI for each job can be opened in a separate window by selecting the job ID. Once there, one can: a. View information about the job, such as name, application ID, start and end times, and duration. b. Review the list of stages involved in the job and examine its progress and any relevant metrics. c. Review the task details, including task IDs, start and end times, and status information. d. Look at the DAG Visualization which shows the logical plans for the job.
Understanding Catalyst Optimizer One of the areas that typically cause confusion is how Catalyst Optimizer works. Catalyst is responsible for taking the SQL, DataFrame, and Dataset code and converting it into physical Spark Spark jobs, but it can be difficult to know whether the SQL or code has been properly optimized until runtime. You can use the EXPLAIN operator to understand how Spark translates queries and determines query plans. The operator provides detailed information about the query optimization. Note that different Spark versions have different EXPLAIN formats.
Debugging Configurations Spark provides a number of configuration options that you can use to debug issues. By configuring certain settings, you can capture more information about what Spark is doing, which may help you understand what's going on under the hood. For instance, you can set the spark.sql.execution.debug parameter to true to enable detailed query execution debugging. This will output relevant debugging information to the Spark logs, including detailed statistics about I/O operations and timing data for each stage.
Using Log Analytics For long running and complex Spark jobs, it can be beneficial to collect and analyze log data. Tools like Azure Log Analytics or Elasticsearch can capture Spark logs and enable analytics on the data. These analytics tools allow you to identify job performance issues, trace data movement through your Spark application, and drill down to investigate job failures or other errors.

Finally, you can use approproiate logging statements and setJobDescription or setLocalProperty with callSite.short and callSite.long. This will help you to log exactly which stage and task is running and you can get the details in the debug logs for each job.