Cannot reserve additional contiguous bytes in the vectorized reader (requested xxxxxxxxx bytes).

shan_chandra
Databricks Employee
Databricks Employee

‎10-11-2021 09:46 AM

I got the below error when running a streaming workload from a source Delta table 

Caused by: java.lang.RuntimeException: Cannot reserve additional contiguous bytes in the vectorized reader (requested xxxxxxxxx bytes). As a workaround, you can reduce the vectorized reader batch size, or disable the vectorized reader, or disable spark.sql.sources.bucketing.enabled if you read from bucket table. For Parquet file format, refer to spark.sql.parquet.columnarReaderBatchSize (default 4096) and spark.sql.parquet.enableVectorizedReader; for ORC file format, refer to spark.sql.orc.columnarReaderBatchSize (default 4096) and spark.sql.orc.enableVectorizedReader

could you please let us know how to mitigate the issue?

shan_chandra
Databricks Employee
Databricks Employee

‎10-11-2021 09:49 AM

This is happening because the delta/parquet source has one or more of the following:

  1. a huge number of columns
  2. huge strings in one or more columns
  3. huge arrays/map, possibly nested in each other

In order to mitigate this issue, could you please reduce spark.sql.parquet.columnarReaderBatchSize from default value - 4096 ?

View solution in original post