Introduction
A single query that runs quickly in the SQL editor may not fully reveal how a warehouse behaves when many users query it concurrently. Under concurrent load, queries queue, contend for resources, and exhibit latency that never appears in isolated runs. Before sizing a warehouse, committing to a latency SLA, or planning a migration, we need measured numbers gathered under realistic concurrent conditions.
This series builds on the basic JMeter test plan introduced in Why Databricks SQL Serverless is the best for BI workloads Part II: Apache JMeter for Databricks SQL, extending it into a repeatable benchmark for Databricks SQL warehouses. Across the three parts, you will be able to:
- Measure query latency and throughput under realistic concurrent load
- Model peak and off-peak traffic within a single test run
- Calculate the exact cost of a benchmark run from Databricks system tables
The series is organized as follows:
- Part 1 (this post): Inputs and Configuration — externalizing SQL and parameters, and separating configuration from the test plan.
- Part 2: Concurrency and the Test Plan Plan — modeling multi-stage load and assembling the JMeter test plan.
- Part 3: Running and Analyzing Results — executing the test and measuring latency and cost.
Building on the Basic Test Plan
The prerequisites for this work — installing Apache JMeter, configuring the Databricks JDBC driver, authenticating with a personal access token, and disabling result caching to measure true compute performance — is covered in Why Databricks SQL Serverless is the best for BI workloads Part II: Apache JMeter for Databricks SQL. Readers who are new to JMeter should review that material first; here we treat the basic test plan as our starting point.
That basic plan does one thing well: it executes a single query against a warehouse and reports the timing. To model a realistic production workload, however, a few additional capabilities are required.
Workload Requirements
Modeling a realistic workload introduces two requirements that a single-query test plan does not address on its own:
- Varied query patterns. When every thread runs identical queries, the result reflects a single path, not the mix of different queries that analysts actually run in production. A representative benchmark must exercise different queries over different parameter values.
- Multi-stage concurrency. Production traffic is not flat. It ramps up, holds at a peak, and ramps down. Validating an SLA across off-peak, steady-state, and peak load therefore requires more than a single fixed concurrency level.
Another requirement is practical rather than about realism: externalized configuration. Connection details, credentials, and the concurrency schedule should live outside the test plan, so that pointing the benchmark at a different warehouse does not require editing the plan itself.
This post addresses the varied query patterns and externalized configuration. Part 2 addresses multi-stage concurrency and builds the test plan itself.
- In practice, this structure lets us model several different benchmark shapes rather than a single fixed load level.
- For example, we can run a flat smoke test to validate the setup, a stepwise business-hours profile that scales up to a peak and then scales back down, or a short spike test that validates behavior during a sudden burst of concurrent queries.
- Part 2 will show how to express these patterns through the concurrency schedule.
Externalizing SQL and Variables
To support varied query patterns, we keep each query in its own file, paired with a CSV of parameter values, rather than hardcoding queries inside the test plan
Create a `/TestPlans/` folder with the following layout, which the test plan expects:
/TestPlans/
/inputs/ ← your SQL files and variable CSVs
/testResults/ ← output CSVs written here at runtime (auto-created)
run.properties ← run configuration (connection, schedule) (later below)
DBSQL_ConcurrentBenchmark.jmx ← the test plan you will build (in Part 2)
The key idea is that queries, parameter values, results, and runtime configuration are kept separate, which keeps the benchmark flexible: changing the queries, parameter values, or target warehouse means editing these files, never the plan itself.
Within `/inputs/`, follow the naming convention `sql<n>.sql` and `variables_sql<n>.csv`:
/inputs/
sql0.sql
variables_sql0.csv
sql1.sql
variables_sql1.csvEach SQL file uses `${variable_name}` placeholders. For example:
SELECT *
FROM samples.nyctaxi.trips
WHERE tpep_dropoff_datetime
BETWEEN ${tpep_pickup_datetime_start}
AND ${tpep_pickup_datetime_end}The matching CSV is semicolon-delimited, with one row of values per iteration:
tpep_pickup_datetime_start;tpep_pickup_datetime_end
'2016-01-01';'2016-01-02'
'2016-01-02';'2016-01-03'
'2016-01-03';'2016-01-04'JMeter gives each thread the next row in the file and starts over at the top once it runs out. So with 3 rows and 10 threads, the values come out as 1, 2, 3, 1, 2, 3, 1, 2, 3, 1. The rows repeat, which is what we want: a small set of parameters is enough to keep every thread busy for a long-running test. For a query that takes no parameters, create a CSV with a single dummy header so that the plan can read it without error:
__dummy__Separating Configuration from the Test Plan
We separate configuration from content: a `run.properties` file holds the connection details and the concurrency schedule, while `/inputs/` holds the queries. Pointing the benchmark at a different warehouse then requires editing only `run.properties`, and changing the set of queries requires editing only the files under `/inputs/`.
# Databricks JDBC connection
user.database.url=jdbc:databricks://<workspace-host>:443/default;httpPath=<http-path>
user.database.jdbc.driver=com.databricks.client.jdbc.Driver
user.database.user=token
user.database.password=<your-personal-access-token>
# Path to your TestPlans folder
user.test.plan.folder=/path/to/TestPlans
# Concurrency profile — explained in Part 2
threads_schedule=spawn(2,0s,0s,60s,0s) spawn(6,60s,0s,60s,0s) spawn(10,120s,0s,60s,0s)
# Limit rows fetched per query, so this remains a compute test rather than a network test
user.test.limit.resultset=1000
# Adds a row_count column (that records rows returned per query - a handy consistency/correctness check across runs) to the results CSV
sample_variables=row_count_# column The server hostname, HTTP path, and personal access token are obtained from the warehouse's Connection details tab. Instructions for generating a token are available in the Databricks documentation.
Summary
In this part, we laid the groundwork for a reusable benchmark: we externalized each query and its parameters, established a naming convention that lets the test plan pick them up automatically, and moved connection details and the concurrency schedule into separate configuration. The result is a clean separation between the test plan and its inputs which matters because a single, version controlled test plan can now be pointed at any warehouse or query mix without edits. That makes benchmarks repeatable, easy to review, and simple to share across teams and projects.
With the inputs and configuration in place, Part 2 explains the multi-stage concurrency profile and builds the JMeter test plan itself — turning these files into a benchmark you can run against any warehouse to measure latency and throughput under realistic, production-like load.
