Cannot reserve additional contiguous bytes in the vectorized reader (requested xxxxxxxxx bytes).

shan_chandra
Databricks Employee
Databricks Employee

‎10-11-2021 09:46 AM

I got the below error when running a streaming workload from a source Delta table 

Caused by: java.lang.RuntimeException: Cannot reserve additional contiguous bytes in the vectorized reader (requested xxxxxxxxx bytes). As a workaround, you can reduce the vectorized reader batch size, or disable the vectorized reader, or disable spark.sql.sources.bucketing.enabled if you read from bucket table. For Parquet file format, refer to spark.sql.parquet.columnarReaderBatchSize (default 4096) and spark.sql.parquet.enableVectorizedReader; for ORC file format, refer to spark.sql.orc.columnarReaderBatchSize (default 4096) and spark.sql.orc.enableVectorizedReader

could you please let us know how to mitigate the issue?