Impact of Capturing Streaming Metrics to ADLS on Data Load Performance

saicharandeepb — Fri, 05 Sep 2025 05:25:24 GMT

Hi Community,

I’m working on capturing Structured Streaming metrics and persisting them to Azure Data Lake Storage (ADLS) for monitoring and logging. To achieve this, I implemented a custom StreamingQueryListener that writes streaming progress data as JSON files using the code snippet below.

To avoid generating multiple small files, I used coalesce(1) to reduce the DataFrame to a single partition so that Spark writes only one output file per batch. While this approach functions as intended, I’ve noticed that writing these metrics—particularly with coalesce(1)—is negatively impacting the overall data load performance.

Has anyone experienced similar performance issues when writing streaming metrics directly to external storage like ADLS?

What are some recommended asynchronous or buffered strategies for capturing and storing streaming metrics without affecting the main data processing workflow?

Re: Impact of Capturing Streaming Metrics to ADLS on Data Load Performance

szymon_dybczak — Fri, 05 Sep 2025 06:25:34 GMT

Hi @saicharandeepb ,

The behaviour you're experiencing can happen with coalesce. The thing is, when you use coalesce(1), you're sacrificing parallelism and everything is performed on a single executor.

There's even a warning in Apache Spark OSS regarding this:

You can also check following posts/blogs:

apache spark - does coalesce(1) the dataframe before write have any impact on performance? - Stack Overflow

(22) Analyzing a 30x Slowdown in My Spark Program Due to Coalesce | LinkedIn

topic Re: Impact of Capturing Streaming Metrics to ADLS on Data Load Performance in Data Engineering

Impact of Capturing Streaming Metrics to ADLS on Data Load Performance

Re: Impact of Capturing Streaming Metrics to ADLS on Data Load Performance