DataFlint on Databricks - the Open Source Spark UI Upgrade Apache Spark Has Needed for Years

[szymon_dybczak]

Introduction

Apache Spark has become one of the most widely adopted engines for large-scale data processing. Its appeal is easy to understand: it supports batch processing, streaming workloads, feature engineering, machine learning pipelines, and large-scale analytical transformations across nearly every major data platform. It gives teams a powerful and flexible way to process data at massive scale.

But that power comes with complexity. Because Spark is a distributed computing engine, its behavior is not always easy to reason about when something stops working as expected. A simple symptom, such as a slow pipeline or a failed job, can be caused by many different things: skewed data, inefficient joins, shuffle bottlenecks, memory pressure, spilling, poor partitioning, executor failures, configuration issues, or an unexpected change in the physical plan.

The Spark UI contains a huge amount of useful information, but making sense of it is not straightforward. Relevant details are scattered across multiple tabs. Each tab gives you part of the picture, but rarely the full story.

The Spark UI itself also feels dated in some areas. When tracking a long-running query, you often need to refresh the page manually to see progress. To understand what is happening, you have to jump between multiple tabs and mentally connect information that is presented separately.
As a result, Spark debugging can become cognitively demanding very quickly. The hard part is not just finding metrics, but understanding which metrics matter, how they relate to each other, and what conclusion can be drawn from them. For many teams, getting from “this job is slow” to “this is the actual bottleneck” still requires a lot of experience, patience, and manual investigation

This is exactly the gap DataFlint aims to close.

What is DataFlint OSS?

DataFlint OSS is an open-source monitoring and debugging plugin for Apache Spark. It does not replace the Spark UI. Instead, it extends it by adding a dedicated DataFlint tab inside each Spark application.

The goal is simple: make Spark performance easier to understand. DataFlint does this by going beyond raw execution data. It highlights patterns that often indicate performance problems, such as data skew, inefficient resource usage, small files, large partitions, problematic joins, or suspicious executor behavior. These findings appear as alerts, helping you move from “something is slow” to a likely explanation much faster.

DataFlint also adds a more modern interface for exploring Spark applications, including run summaries, stage breakdowns, heat maps, syntax highlighting, and optional instrumentation for detailed operator-level timing. We will look at these features in the demo section, but the key idea is this: DataFlint makes it much easier to understand what happened in a Spark job and where to focus your attention.

How it works?

DataFlint is built on top of Apache Spark’s official plugin API. When a Spark application starts, Spark loads SparkDataflintPlugin through the normal plugin lifecycle. The plugin then creates a driver-side component, SparkDataflintDriverPlugin, and Spark calls its init() method during driver startup, before any jobs begin running.

This is important - DataFlint uses native extension mechanism Spark provides for instrumentation and runtime integrations.

The initialization flow has two main steps:

1. First, during init(), DataFlint can register a SQL extension called DataFlintInstrumentationExtension. This only happens when at least one instrumentation option is explicitly enabled. When enabled, the extension modifies Spark SQL physical plans by wrapping selected operators with timing nodes. These wrappers collect wall-clock duration metrics for parts of the query plan that the native Spark UI does not expose in the same level of detail.
2. Second, after init() completes, Spark calls registerMetrics(). At this point, DataFlint installs its Web UI tab, registers the REST endpoints used by the frontend, serves the bundled React single-page application, and attaches event listeners to Spark’s listener bus.

The result is a modern single-page application embedded directly into the existing Spark Web UI. It can poll live application metrics, update without full page refreshes, and run without requiring a separate service outside the Spark driver process.

Installation

There are two main ways to install DataFlint on Databricks:

1. Install it directly from a notebook (super easy):
   https://dataflint.gitbook.io/dataflint-for-spark/getting-started/install-on-databricks#install-on-da...
2. Install it as a Spark plugin on a Databricks cluster, which is the recommended approach.

Since the second option is recommended, let’s walk through that setup.

Before creating the init script, we need a location where the script can be stored. Databricks allows us to use either a Unity Catalog volume or workspace files for this purpose.

In this example, we will use a Unity Catalog volume for two reasons. First, Unity Catalog volumes are the recommended option when using DBR 13.3 LTS or later with Unity Catalog enabled. Second, workspace files are not supported on clusters running in standard access mode( formerly known as shared access mode).

To create a volume, we can use the following command:

%sql
CREATE VOLUME databricks_demo_ws.default.demo_volume;

Once the volume is ready, the next step is to prepare the init script. Open the DataFlint documentation and copy the init script that matches your Spark version. In my case, the cluster runs on Apache Spark 4.0, so I selected the corresponding DataFlint script.

Save the script content as init_script.sh, and upload it to the Unity Catalog volume.

One important thing to keep in mind is that standard access mode requires an administrator to add init scripts to the allowlist. Without this step, the cluster will fail to start and return an error similar to the following:

To handle this, open Unity Catalog Explorer and follow these steps:

1. In your Databricks workspace, click Catalog.
2. Click the gear icon.
3. Click the metastore name to open the metastore details and permissions page.
4. Select Allowed JARs/Init Scripts.
5. Click Add init script and provide the correct path to your script.

Once the init script has been added to the allowlist, go to your compute configuration. Open the Advanced section, then go to Init scripts. Choose Volumes as the source and provide the full path to the uploaded init_script.sh file.

Great, at this step we’re ready for installation. During my first attempt, the cluster failed to start because of a small typo in the init script that was available in the DataFlint documentation at the time.

After a short debugging session, I corrected the script locally and flagged the issue to the DataFlint team. They responded very quickly, and the documentation issue has already been fixed.

I’m leaving this note here for transparency, and also as a reminder that when working with init scripts, even a very small typo can prevent the cluster from starting.

Below is the corrected version you can copy and paste:

DATAFLINT_VERSION="0.9.9"
SPARK_DEFAULTS_FILE="/databricks/driver/conf/00-custom-spark-driver-defaults.conf"

mkdir -p /databricks/jars/

wget --quiet \
  -O /databricks/jars/dataflint_spark4-databricks_2.13-$DATAFLINT_VERSION.jar \
  https://repo1.maven.org/maven2/io/dataflint/dataflint-spark4-databricks_2.13/$DATAFLINT_VERSION/dataflint-spark4-databricks_2.13-$DATAFLINT_VERSION.jar

if [[ $DB_IS_DRIVER = "TRUE" ]]; then
  mkdir -p /mnt/driver-daemon/jars/
  cp /databricks/jars/dataflint_spark4-databricks_2.13-$DATAFLINT_VERSION.jar /mnt/driver-daemon/jars/dataflint_spark4-databricks_2.13-$DATAFLINT_VERSION.jar
  echo "[driver] {" >> $SPARK_DEFAULTS_FILE
  echo "  spark.plugins = io.dataflint.spark.SparkDataflintPlugin" >> $SPARK_DEFAULTS_FILE
  echo "}" >> $SPARK_DEFAULTS_FILE
fi

If you author init_script.sh on a Windows machine, your editor will very likely save it with Windows (CRLF) line endings (\r\n). The Databricks driver runs Linux, where the shell expects Unix (LF) line endings (\n).
A script saved with CRLF will fail in confusing ways - the trailing carriage return gets attached to the last token on each line, so you'll see errors such as:

/bin/bash: line 2: $'\r': command not found

How to avoid it:

• In VS Code, click the CRLF indicator in the bottom-right status bar and switch it to LF, then save.

This single issue comes up surprisingly often in Databricks Community threads, so I think it’s worth mentioning here.

DataFlint Demo: Seeing It in Action

Once the installation is complete, we can take a closer look at what DataFlint actually adds to the Spark debugging experience.

To make the walkthrough more concrete, let’s run a sample analytical query against the TPC-DS dataset and then open the DataFlint tab in the Spark UI:

SELECT
    w.w_warehouse_name,
    it.i_category,
    it.i_class,
    COUNT(DISTINCT i.inv_item_sk) AS distinct_items,
    COUNT(DISTINCT i.inv_date_sk) AS inventory_dates,
    COUNT(*) AS inventory_rows,
    SUM(i.inv_quantity_on_hand) AS total_quantity_on_hand,
    AVG(i.inv_quantity_on_hand) AS avg_quantity_on_hand,
    MIN(i.inv_quantity_on_hand) AS min_quantity_on_hand,
    MAX(i.inv_quantity_on_hand) AS max_quantity_on_hand
FROM samples.tpcds_sf1000.inventory i
JOIN samples.tpcds_sf1000.warehouse w
    ON i.inv_warehouse_sk = w.w_warehouse_sk
JOIN samples.tpcds_sf1000.item it
    ON i.inv_item_sk = it.i_item_sk
GROUP BY
    w.w_warehouse_name,
    it.i_category,
    it.i_class
ORDER BY
    total_quantity_on_hand DESC;

For the demo, we will follow a typical investigation flow: start with the high-level symptoms, check whether the cluster was used efficiently, inspect the SQL plan, let alerts point us to likely issues, and only then enable deeper instrumentation if we need more precise timing.

1. Start with the Summary page to understand the workload at a high level.
2. Check cluster resources to see how executors, memory, and cores behaved during the run.
3. Inspect the SQL plan to identify expensive operators and heavy parts of the query.
4. Use Alerts to jump directly to suspicious patterns instead of manually hunting through metrics.
5. Enable instrumentation when you need more precise operator-level timing.

Step 1: Start with the Summary Page

The Summary page gives a high-level view of how your query used cluster resources during execution. Instead of immediately jumping between Spark UI tabs such as Jobs, Stages, Executors, and SQL, DataFlint brings the most important performance indicators into one place.

This makes the Summary page a useful first checkpoint. It helps you quickly understand whether a Spark job was efficient, over-provisioned, memory-constrained, shuffle-heavy, or affected by spill operations.

At the top of the page, DataFlint shows metrics such as Duration, DCU, Input, Output, Memory Usage, Shuffle Read, Shuffle Write, Spill to Disk, Idle Cores, and Task Error Rate. Together, these metrics describe both the cost and behavior of the workload.

One metric that deserves a short explanation is DCU, which stands for DataFlint Compute Units. DCU is DataFlint’s measurement unit for Spark usage, similar in concept to a Databricks Unit, or DBU. It combines CPU and memory allocation into a single usage metric.

The formula is:

DCU = (Core/Hour usage * 0.05) + (GiB Memory/Hour usage * 0.005)

Core/Hour: is the number of cores allocated for your app in hours measurement.

GiB Memory/Hour: is the number of memory in GiB units allocated for your app in hours measurement

Another useful detail is that the Summary page refreshes automatically in real time. Unlike the standard Apache Spark Web UI, you do not need to manually refresh the page to see updated metrics while the application is running. This makes live monitoring much more comfortable, especially when you want to watch resource usage, memory pressure, shuffle volume, spill, or task failures as they happen.

Step 2: Check whether the cluster was used efficiently

Next, let’s look at the Resources page. This page gives a more focused view of how Spark resources are allocated and used during the job.

The Executors Timeline makes dynamic allocation easy to understand visually. In this run, Spark started with 1 executor, and later scales to 2 executors. The configuration table below the chart shows the executor and driver resources, such as cores and memory, together with dynamic allocation settings like minimum and maximum executors.
DataFlint puts the resource timeline and configuration details in one place, which makes it easier to connect workload behavior with cluster behavior.

Step 3: Inspect the SQL Plan

After checking the overall workload and cluster behavior, we can move from symptoms to execution details. The Summary page contains a list of executed and currently running queries. When you click one of them, DataFlint opens an interactive SQL execution plan graph.

For example, clicking query ID 10 opens a graph view of the query plan. The plan can be viewed in three modes:

• I/O Only: input/output scan and write nodes.
• Basic: the main transformations.
• Advanced: every node in the plan.

Each node shows the operator name, such as Filter, Exchange, or FileScan, together with key metrics like output rows, shuffle bytes, spill size, partitions, and table name for scans.

A really useful feature is the performance heat bar at the top of each node. It uses green, orange, and red to show how much of the total query time was spent in that operator.

There is also a MiniMap in the lower-left corner that directly complements the heat map. While the main graph lets you zoom in on individual nodes, complex queries can have plans that are too large to view in full at once. The MiniMap gives you a bird’s-eye view of the entire plan, with the same heat map coloring applied, so red nodes remain visible even when they are off-screen.

Together, the heat map and MiniMap let you quickly locate the most expensive operator - and that’s super cool.

Some nodes can also display additional badges:

• A green flag badge indicates that the node is instrumented by TimedExec and has precise wall-clock timing.
• A rocket badge indicates that the operator is running on a native accelerator such as Gluten, Comet, or Photon.
• An alert badge indicates that DataFlint detected a potential issue on that node.

Shuffle operations are especially nice in this view. DataFlint splits Exchange nodes into two separate half-nodes: shuffle write and shuffle read. Each side has its own metrics and stage association, which makes it much easier to see where shuffle cost is coming from.

The plan view also includes a stage grouping toggle. When enabled, nodes that run in the same Spark stage are visually enclosed in a stage container. Clicking a stage opens a side drawer with stage-level details.
This is another feature with no equivalent in Spark’s native UI. In Apache Spark, a query’s execution plan and its stages are two completely separate concepts - the SQL tab shows you the plan tree, and the Stages tab shows you a flat list of stages, with no visual connection between them. DataFlint bridges this gap by overlaying stage boundaries directly onto the plan graph.

Finally, there is a duration mode toggle. You can switch between exclusive duration, which means time spent in that operator only, and inclusive duration, which means time spent in the operator and all of its children.

The bottom toolbar contains useful navigation shortcuts:

• Speed: cycles through nodes ordered by duration percentage, highest first.
• Warning: cycles through nodes with alerts.
• Storage: cycles through nodes with spill, ordered by spill size.
• Fit view / zoom controls: help you navigate large plans.

Step 4: Let Alerts Point to the Problem

So far the Summary and Resources pages have helped us observe what a job did. The Alerts page is where DataFlint goes a step further and starts reasoning about it. Instead of leaving you to interpret the metrics yourself, it continuously inspects every query, stage, and executor against a set of built-in heuristics and surfaces the ones that look wrong - each one written up as a short, plain-English finding with a concrete suggestion on how to fix it.

This is probably the single biggest difference from the native Spark UI. The native UI will show you that one task in a stage ran for 28 seconds while the others finished in a second but it will never tell you “this is data skew, and here’s what to do about it.”

You need to know where to look, which numbers to compare, and what the comparison means. The Alerts page encodes that experience for you.

When you open the tab, alerts are grouped by type and tagged as either a warning (yellow) or an error (red), with a running count of each at the top. Every alert card has a “Go to alert” button that jumps straight to the exact SQL node, stage, or resource the finding is about, so you never have to hunt for where the problem lives.

A concrete example: data skew

The TPC-DS sample tables I’ve used so far are nicely balanced, so they will not show a strong skew problem on their own. To make the issue visible, we need to manufacture one. For the synthetic example, imagine a multi-tenant event platform where most tenants are small, but one enterprise tenant generates almost all of the traffic. This is something you can encounter in real systems: one customer, account, region, or tenant dominates the data distribution.

The setup is simple. Most tenants produce only a small number of events and have one routing rule. But `tenant_enterprise_001` produces around 97% of all events and has 100 routing rules. Then we run a reasonable-looking analytics query: join events to routing rules and summarize how many events each rule matched.

Before running the query, I also move a few Spark optimizations out of the way. I disable broadcast joins so Spark cannot simply broadcast the rules table and avoid the shuffle. I also disable Adaptive Query Execution so Spark’s automatic skew handling does not rescue the query before DataFlint has anything interesting to show.
In production, AQE is usually something you want enabled. For a demo like this, though, turning it off lets the skew surface clearly.

from pyspark.sql import functions as F

spark.conf.set("spark.sql.adaptive.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "200")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
spark.conf.set("spark.sql.join.preferSortMergeJoin", "true")


N = 10_000_000

events_df = (
    spark.range(0, N, 1, numPartitions=128)
    .select(
        F.when(F.rand(seed=42) < 0.97, F.lit("tenant_enterprise_001"))
         .otherwise(
             F.concat(
                 F.lit("tenant_"),
                 F.lpad((F.col("id") % 20_000).cast("string"), 5, "0")
             )
         ).alias("tenant_id"),

        F.col("id").alias("event_id"),
        F.concat(F.lit("session_"), (F.col("id") % 2_000_000).cast("string")).alias("session_id"),
        F.sha2(F.col("id").cast("string"), 256).alias("payload")
    )
)

events_df.createOrReplaceTempView("demo_events")

normal_rules_df = (
    spark.range(1, 20_001, 1, numPartitions=32)
    .select(
        F.concat(
            F.lit("tenant_"),
            F.lpad(F.col("id").cast("string"), 5, "0")
        ).alias("tenant_id"),
        F.lit(1).alias("rule_id"),
        F.lit("standard_rule").alias("rule_type")
    )
)

enterprise_rules_df = (
    spark.range(1, 101, 1, numPartitions=4)
    .select(
        F.lit("tenant_enterprise_001").alias("tenant_id"),
        F.col("id").alias("rule_id"),
        F.concat(F.lit("enterprise_rule_"), F.col("id").cast("string")).alias("rule_type")
    )
)

rules_df = normal_rules_df.unionByName(enterprise_rules_df)

rules_df.createOrReplaceTempView("demo_routing_rules")


skew_query = """
SELECT
    r.rule_type,
    COUNT(*) AS matched_events,
    COUNT(DISTINCT e.session_id) AS unique_sessions,
    MIN(e.event_id) AS first_event_id,
    MAX(e.event_id) AS last_event_id
FROM demo_events e
JOIN demo_routing_rules r
  ON e.tenant_id = r.tenant_id
GROUP BY r.rule_type
ORDER BY matched_events DESC
"""

display(spark.sql(skew_query))

After the query finishes, open the Alerts page in DataFlint. You should see a Partition Skew warning for the join stage.

If you click Go to Alert, you will be redirected straight to the location in the physical plan where the problem occurs. This is a super nice feature.

It is worth to explain why DataFlint treats this as skew rather than normal variation. The alert is based on the task-duration distribution within a stage. DataFlint compares the slowest task with the median task and raises a warning only when the difference is large enough to matter. That avoids noisy alerts for tiny stages where one task being a bit slower is irrelevant.

In this example, the warning is expected because the workload is intentionally unbalanced. The median task processes a small tenant-sized partition, while the worst task processes the enterprise tenant and its 100-rule fan-out. That is exactly the scenario skew alerts are meant to make obvious.

Another common offender: small files

Skew is about time; the next example is about layout. The problem here is a table written as thousands of tiny files instead of a few large ones - when you read it back, Spark spends more effort opening files and scheduling tasks than actually processing data. We can reproduce it intentionally by spreading a small amount of data across far too many output files:

-- Force 5,000 output files for a tiny dataset -> a few hundred rows
-- (a few KB) per file.
CREATE TABLE default.tiny_files AS
SELECT /*+ REPARTITION(5000) */ id, rand() AS v
FROM range(0, 1000000); 

A million rows split across 5,000 files is roughly 200 rows - a few kilobytes per file. Simply reading the table back triggers the Reading Small Files warning:

SELECT SUM(v)
FROM default.tiny_files;

The heuristic here is simple: DataFlint divides the bytes read by the number of files read for a scan, and if the average file is smaller than a few megabytes and the scan touched more than a hundred files, it flags it. There’s a matching alert on the write side too - if your job is the one producing the small files, DataFlint will point that out and even tailor the advice to whether the output is partitioned.

Step 5: Experimental feature: DataFlint Spark Instrumentation

DataFlint provides optional instrumentation that enhances Spark observability. It injects extra metrics and metadata into the Spark UI that Spark does not expose on its own. All instrumentation is opt-in and disabled by default, so nothing about your query planning changes unless you explicitly turn it on.

The mechanism is worth understanding before we use it. When any instrumentation flag is enabled, DataFlint registers a Spark SQL extension during driver startup. That extension hooks into Spark’s physical planning phase and wraps selected operators with a lightweight timing node. The wrapper is transparent - it shows up as a single node in the plan graph (its name simply gets a DataFlint prefix), it keeps all of the operator's original metrics, and it adds one new metric: duration, the wall-clock time that operator actually spent doing work.

Instrumentation is split into granular flags so you can enable just what you need. The two we’ll look at here are Window instrumentation and SQL nodes instrumentation:

# enable only window timing
.config("spark.dataflint.instrument.spark.window.enabled", "true")

# enable timing for the common SQL operators (filters, joins, scans, aggregates, ...)
.config("spark.dataflint.instrument.spark.sqlNodes.enabled", "true")

# or turn everything on at once
.config("spark.dataflint.instrument.spark.enabled", "true")

On Databricks you’d add the same keys to your cluster’s Spark config.

Window instrumentation

Window instrumentation wraps Spark’s WindowExec so you can see how long the window computation took, right on the plan node.

Let’s try it with what below dummy query:

SELECT
    c_customer_sk,
    c_last_name
FROM samples.tpcds_sf1.customer
QUALIFY row_number() OVER (PARTITION BY c_last_name ORDER BY c_customer_sk DESC) = 1;

We enable spark.dataflint.instrument.spark.window.enabled, run the query, open the SQL plan... and, weirdly, there is no window operator to be found. What happened?

After some investigation it turns out this query never uses a plain WindowExec operator at all. Because the window is immediately filtered by QUALIFY, Spark applies an optimization introduced in [SPARK-37099] - a dedicated physical operator called WindowGroupLimit.

The idea behind SPARK-37099 is as follows: for rank-style functions such as row_number, rank, and dense_rank, the rank of a key computed on a partial dataset is always less than or equal to its final rank over the full dataset. This means Spark can safely discard rows whose partial rank already exceeds k, before the expensive shuffle and window processing take place. To do this, Spark inserts a per-window-group limit both before and after the shuffle.

As a result, the window execution time we intended to measure is now captured inside WindowGroupLimit, which is not covered by the window flag.

Let’s remove the filtering from our query.

SELECT
    c_customer_sk,
    c_last_name,
    row_number() OVER (PARTITION BY c_last_name ORDER BY c_customer_sk DESC)
FROM samples.tpcds_sf1.customer

After that, we should be able to see the window instrumentation. Great - we’re learning DataFlint and Spark internals at the same time! And as you can see -> Window instrumentation worked this time.

And good news: I asked the DataFlint team about WindowGroupLimitExec support, and they confirmed that it is already planned for the next release. So soon we should have instrumentation for this operator as well - nice.

SQL nodes instrumentation

SQL nodes instrumentation casts a much wider net. Instead of a single operator family, it wraps the common physical operators that make up most query plans - filters, projections, joins, sorts, hash/sort aggregates, the file and batch scans, the write command.

When enabled, DataFlint’s DataFlintInstrumentationExtension wraps each SQL physical operator with a TimedExec node that measures actual wall-clock execution time per operator. The result is a duration metric on every
instrumented node - not an estimate derived from task metrics, but a direct measurement of how long that specific operator spent processing data. This is what powers the heat map: without instrumentation, duration percentages are approximated from stage-level data; with instrumentation, every node carries its own precise timing.

On the screen below, you can see that after enabling SQL node instrumentation, wrapped operators are visible even in the native UI.

The practical impact is significant. Consider a stage containing a SortMergeJoin followed by a Filter followed by a Project. At the stage level, they all look the same — part of a 3-minute stage. With instrumentation, you might discover the join consumed 2 minutes 50 seconds, and the filter ran in under a second. That distinction is the difference between tuning the right thing and tuning the wrong thing.

Instrumentation is intentionally opt-in - it rewrites the physical query plan, which carries a small overhead and a compatibility risk with native accelerators like Gluten or Comet. But for workloads where you need to understand performance at operator granularity rather than stage granularity, it transforms the plan view from a structural diagram into a genuine performance profile.

Conclusion

DataFlint is an excellent addition to the Spark ecosystem. It makes day-to-day debugging much easier by bringing the most important pieces of information into one place, highlighting suspicious patterns, and helping you move from “something is slow” to “this is probably why” much faster.

I have already started using DataFlint in my daily workflow, and it has made Spark performance investigation feel much less painful. If you work with Spark regularly, I definitely recommend giving it a try.

You can check out the project on GitHub, and if you find it useful, consider giving the repository a star. It’s a simple way to support the project and help more Spark users discover it.

Also, if you are interested in Spark optimization more broadly, I highly recommend checking out the DataFlint YouTube channel and the Big Data Performance Substack, which is run by one of DataFlint’s founders. Both are great resources for learning more about Spark performance, debugging, and optimization.

And finally, if you want to help improve the Spark debugging experience for everyone, consider contributing to the project.