Databricks Community

susmitsircar

We are thinking to change the Spark configuration for Parquet files to use zstd compression.Configuration: spark.sql.parquet.compression.codec = zstdThis will only affect new data written by our Spark jobs. All existing data will remain compressed wi...

susmitsircar · 08-12-2025

We are encountering an issue in our Spark streaming pipeline when attempting to write checkpoint data to S3. The error we are seeing is as follows:25/08/12 13:35:40 ERROR RocksDBFileManager : Error zipping to s3://xxx-datalake-binary/event-types/chec...

susmitsircar · 07-28-2025

I'm encountering the following error while trying to upload a RocksDB checkpoint in Databricks:java.lang.IllegalStateException: Found no SST files during uploading RocksDB checkpoint version 498 with 2332 key(s). at com.databricks.sql.streaming.s...

susmitsircar

Yes my believe is it should support 7.3 LTS as well, we will prove it with thorough testingThanks for the discussion. Cheers

susmitsircar

zstd Parquet support is not available for Databricks Runtime 7.1 LTS and other 7.x versions; these versions only support codecs like snappy, gzip, and lzo. Enabling zstd for Parquet in DBR 7.1 LTS will result in compatibility issues—jobs may fail to ...

susmitsircar

but I can see zstd-jni in the release noteshttps://docs.databricks.com/aws/en/archive/runtime-release-notes/7.3lts

susmitsircar

Thanks for the reply @ManojkMohan As far as I know from Spark 3.0.0 zstd is supported, so ideally any DBR >= 7.3 LTS (which uses Apache Spark 3.0.1) should work with zstdIs my understanding wrong here?

susmitsircar · 08-12-2025

Thanks for your reply i don't found this parameter in public docs, can you guide me where can I findspark.conf.set("spark.sql.streaming.stateStore.rocksdb.cleanupCheckpoint", "true")

Databricks Community

User Stats

User Activity

Proposal: Switch to Zstd Compression for Parquet to Reduce S3 Costs

Spark streaming failing intermittently with FileAlreadyExistsException RocksDB checkpointing

Spark streaming failing intermittently with llegalStateException: Found no SST files

Re: Proposal: Switch to Zstd Compression for Parquet to Reduce S3 Costs

Re: Proposal: Switch to Zstd Compression for Parquet to Reduce S3 Costs

Re: Proposal: Switch to Zstd Compression for Parquet to Reduce S3 Costs

Re: Proposal: Switch to Zstd Compression for Parquet to Reduce S3 Costs

Re: Spark streaming failing intermittently with FileAlreadyExistsException RocksDB checkpointing