Load Testing Databricks SQL Warehouses with JMeter — Part 3: Running and Analyzing Results

[DineshBabuK]

Introduction

This is the final part of a three-part series on concurrent load testing of Databricks SQL warehouses with Apache JMeter. In Part 1 we externalized our SQL queries and parameters, and in Part 2 we configured the multi-stage concurrency profile and built the JMeter test plan. In this part, we run the benchmark and analyze the results — measuring latency per query and calculating the exact cost of the run from Databricks system tables.

Running the Test

With the test plan built and `run.properties` configured, the benchmark runs from the command line. The `-n` flag disables the GUI, which would otherwise consume resources and skew latency measurements, and `-q` loads configuration from the properties file:

jmeter -n -q run.properties -t DBSQL_ConcurrentBenchmark.jmx

Because the Summary Report listener writes to the results path from `run.properties`, each run produces a timestamped CSV under `/testResults/`. JMeter records one row per query execution: the label column holds the SQL filename — this is simply the JDBC sampler's name, which we set to the .sql filename in Part 2, so each query is identifiable in the results — and the elapsed column gives that query's execution time in milliseconds.

Analyzing the Results

Performance metrics. JMeter already produces everything needed to analyze latency. Open the timestamped results CSV from `/testResults/` in JMeter and generate the HTML dashboard from the same file with (the -o directory must be new or empty, or JMeter refuses to write the report):

jmeter -g testResults_<MM-dd-yyyy-HH-mm-ss>.csv -o ./benchmark-report

The dashboard reports each .sql file under its own label, so you get per-query latency, not one combined number. In the Statistics table, use the median (P50) for typical performance and the 95th percentile for the slow tail an SLA cares about (it ignores rare slow runs). The 99th percentile shows the worst cases like queuing or cold starts. Ignore the Average - a few slow runs pull it upward.

Figure 1: The JMeter dashboard's Statistics table — one row per query (sql0–sql3) plus a Total, giving sample count, average, median, 90th/95th/99th percentiles, max, error rate, and throughput. Judge latency by the median and percentiles rather than the average.

This run was clean (this example uses the short concurrency profile, not the full 8-hour business-day run): 3,254 queries, 0 errors, and all four SQL files land close together (medians 449–537 ms). The Total row's median (491 ms) and mean (517 ms) are nearly identical, so there's no long tail skewing the numbers — 95% of queries finish under 647 ms, 99% under 1.1 s, with the slowest single query at 3.2 s. Against a sub-second P95 target, this warehouse clears it comfortably.

The Statistics table shows how each query performed overall, but not how the warehouse held up as concurrency rose and fell. For that, the dashboard plots the run over time. Read each graph against the thread count.

Active Threads Over Time. The load shape itself — the multi-stage profile from Part 2 as it actually ran, ramping up, holding, and ramping down. Everything below should be read against this curve.

Figure 2:  Active threads over the run — the multi-stage concurrency profile from Part 2 as it was actually executed.

Response Times Over Time. Query time as the run progresses. If the line stays flat while threads climb, the warehouse is absorbing the load; a rise at the peak is where queuing starts. Here it stays flat — the added concurrency didn't push query times up.

Figure 3: Response time over the run. Lower is better.

Response Time Percentiles Over Time. The best graph for an SLA discussion, since it shows P50/P90/P95/P99 evolving across stages. A struggling warehouse keeps a flat median while P95 and P99 climb at the peak — the tail the summary table's single P95 hides. Here the bands stay close and level.

Figure 4: Response-time percentiles (P50, P90, P95, P99) over the run. Lower is better.

Transactions Per Second. Throughput — how many queries per second the warehouse completed. This run sustained about 13.5 q/s throughout. When the requirement is a target throughput rather than a latency ceiling, this is the graph to watch; it should hold steady through the peak stages.

Figure 5: Throughput in queries per second over the run.

💡  Tip — Analyze Results with Genie. 

Load the CSV results into a table (e.g. benchmark_results) and point a Genie space at it. You can then ask in plain language: "What's the P95 latency per SQL file?", "Which stage had the highest median response time?", "How did throughput change as active threads increased?" And Genie returns the query and a chart, no manual aggregation needed as shown below.

Cost. Latency alone doesn't decide warehouse sizing — cost does too. So measure each run by both its elapsed time and the DBUs it consumed. Note the run's start and end timestamps and query the system tables:

Prefer a UI view? The same figures are available in the Account Console Cost dashboard (or a AI/BI dashboard over system.billing.usage); the query below gives the exact per-window number for a benchmark run.

SELECT
    u.usage_metadata.warehouse_id,
    u.usage_quantity,
    p.pricing.effective_list.default                     AS unit_price,
    u.usage_quantity * p.pricing.effective_list.default  AS total_cost
FROM system.billing.usage u
JOIN system.billing.list_prices p ON u.sku_name = p.sku_name
  AND u.usage_end_time >= p.price_start_time
  AND (p.price_end_time IS NULL OR u.usage_start_time <= p.price_end_time)
WHERE u.usage_metadata.warehouse_id = '<your-warehouse-id>'
  AND u.usage_end_time   >= '<test_start_time>'
  AND u.usage_start_time <= '<test_end_time>'
  AND u.usage_unit = 'DBU'
ORDER BY u.usage_start_time;

--Note: Confirm sku_name / usage_unit resolves to the warehouse compute SKU only;

For the Medium-warehouse run compared later, this returns one row: about 6.0 DBUs at ~$0.70/DBU, so ~$4.18 for the window.

One gotcha: system.billing.usage is not real-time. Usage records land minutes to a few hours later, so a query run right after the benchmark may return zero or partial rows — wait until the window is fully populated before trusting the total. This lag is also why the cost step stays separate from the JMeter plan rather than running as a teardown query.

For repeated benchmark runs, record this cost alongside the latency summary so that different concurrency profiles can be compared not only by performance, but also by cost efficiency.

Interpreting Benchmark Cost

Cost for a single run is useful, but the value comes from comparing runs. Two different questions matter here, and they matter to different people:

• How fast is the query? — the workload owner's concern, answered by P50 and P95 latency.
• What does the profile cost to run? — the platform or finance owner's concern, answered by total DBUs and total cost.

These are not the same axis, and plotting P95 latency straight against total run cost can mislead: a bigger warehouse finishes work faster but, over the same time window, draws more DBUs — so it looks both faster and more expensive at once. To compare them fairly, normalize cost to the work done — cost per 1,000 queries — so both numbers describe the same unit. Total cost still answers "what did the run cost," but the sizing tradeoff belongs on the per-query figure.

The goal isn't the cheapest run — it's the lowest-cost configuration that still meets the target latency or throughput. A larger warehouse may cut queuing and improve response times, but if the gain is small next to the extra DBUs, the smaller one is the better operational choice.

Figure 6 below shows a simple example of how to compare benchmark runs across warehouse sizes for the same scenario.

Figure 6: Example comparison of benchmark runs across warehouse sizes, showing P50 and P95 latency alongside cost per 1,000 queries for the same concurrency scenario. Lower is better for both P95 latency and cost per query.

💡 Generate Visual with an LLM 

You can generate a comparison like this from your own run summaries with an LLM prompt such as: "I ran the same JMeter concurrency profile against Small, Medium, and Large SQL warehouses. Here are the P50/P95 latencies, total DBUs, and total cost for each. Build a table with a normalized cost-per-1,000-queries column and a bar chart comparing P95 latency against cost per 1K queries, and flag which runs meet a P95 ≤ 1,000 ms SLA.”

Against a target SLA of a P95 at or below 1,000 ms for the business-hours profile, the "Meets SLA?" column evaluates each run. Because each run executes a fixed-duration profile rather than a fixed number of iterations, larger warehouses show lower latency but higher total DBUs, sustaining more capacity and doing more work over the same window. If iterations were fixed instead, a faster warehouse would finish sooner, yielding similar total costs across sizes. This comparison reflects real-world business-day sizing by evaluating the same workload profile over the same period.

Scenario	Warehouse	P50 Latency (ms)	P95 Latency (ms)	Total DBUs	Total Cost	Cost / 1K queries	Meets SLA?	Notes
Business-hours profile	Small	520	980	3.8	$2.66	$0.82	Yes	Lowest cost, weaker P95
Business-hours profile	Medium	410	720	6.0	$4.20	$0.79	Yes	Balanced latency and cost
Business-hours profile	Large	360	650	8.7	$6.09	$0.88	Yes	Lowest latency, highest cost

This keeps the cost discussion tied to the goal: choosing the warehouse that fits both performance and cost.

The values in Figure 6 and Table above are illustrative examples. Replace them with results from your own runs.

Best Practices

• Validate SQL files before a long run. A failing query does not stop the run — JMeter records it as an error in the results file rather than halting — so a broken .sql file can quietly skew a whole benchmark. Execute each `.sql` file once in the Databricks SQL editor before running an extended benchmark.
• Run JMeter close to the workspace. Network round-trip time inflates latency measurements, so run JMeter from a machine in the same cloud region as the warehouse.
• Decide how result caching should be handled, and document the choice. Earlier in this series we disabled result caching to isolate raw compute performance — that was specifically to measure *cold* execution. More generally, though, caching is not something to disable by default — the right choice depends on what you are measuring. To reproduce warm, cache-hit performance (typical for repeated BI dashboards), leave caching on; to measure cold execution, turn it off by setting use_cached_result=false as a parameter on the JDBC connection string so it applies consistently across every thread. Whichever mode you choose, record it with the results, since it materially affects latency. Why Databricks SQL Serverless is the Best for BI Workloads gives useful background.
• Benchmark the warehouse type you actually plan to run. The right choice is driven by requirements, not a blanket preference: benchmark Serverless if that is what you will deploy, and Classic or Pro if that is your target. Just keep the type consistent across compared runs, and note that Serverless and Classic differ in cold-start behavior and scaling, so results are only comparable within the same type.
• Parameterize the run, don't hard-code it. Keep warehouse URL, credentials, query set, concurrency schedule, and result paths in run.properties so the same plan can be pointed at a different warehouse or load profile by editing configuration alone.
• Run benchmarks from the command line, not the GUI. Use the GUI only to build and debug the plan; run actual measurements in non-GUI mode (`jmeter -n`), as noted under Running the Test.
• Give JMeter enough capacity. A load generator that is itself CPU- or memory-starved will report inflated latency, so run JMeter on an adequately sized VM so the client is never the bottleneck.

Conclusion

Across this series we turned a basic JMeter test plan into a reusable benchmark that answers the two questions that actually drive a warehouse decision: how fast is it, and what does that speed cost? The payoff is self-service. Once the plan is built, anyone can point it at a different warehouse or load profile by changing configuration alone — no rebuilding — and every run produces the same kind of evidence: per-query latency (P50/P95) alongside the DBU consumption and dollar cost of the run.

Whether the goal is validating a performance SLA, sizing a warehouse for a new workload, or establishing a baseline ahead of a migration, the benchmark replaces guesswork with measured latency and cost figures — so you can choose the smallest, cheapest configuration that still meets the target, and back the choice with numbers.