Databricks Community

VaderKB · 07-22-2025

Hello,I am running a spark streaming job that reads data from AWS Kinesis and writes data to extrenal delta tables which are stored in S3. But I have noticed that over the time, the latency has been increasing. I also noticed that for each batch, the...

VaderKB · 07-29-2025

@Khaja_Zaffer , Thank you for your answer. Yes this is in production environment. Doing an OPTIMIZE did reduce the latency. But I don't understand why? Because from what I understand, OPTIMIZE compacts the file into larger files. And from what I read...

VaderKB · 07-22-2025

@Khaja_Zaffer Sorry the "But" is a typo. The query is simple: I read using readstream, extract the data and expand the json into a table structure and write the data back using writestream with append mode ,checkpoints and mergeSchema as True. It is ...

Databricks Community

User Stats

User Activity

Does too many parquet files in delta table impact writes for the streaming job

Re: Does too many parquet files in delta table impact writes for the streaming job

Re: Does too many parquet files in delta table impact writes for the streaming job