Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2023 12:08 AM
We've got the following info from databricks support which might be of interest for you:
1. “Received message larger than max” error for files with row size smaller than 128MB (For example parquet file you provided)
- We were able to mitigate this issue by adjusting maxRecordsPerBatch.
- The engineering team has prepared and merged a fix which should resolve this issue.
- After the fix is deployed, any file containing a row size smaller than 128 MB will not receive this error.
- This fix will be part of our next maintenance release (The next tentative maintenance is scheduled between the 23rd to 29th Oct in multiple stages. Meanwhile, you can use maxRecordsPerBatch config as a mitigation.
2. “Received message larger than max” error for files with row size larger than 128 MB. (For example, the binary file where rows are > 128 MB)
- We do not officially support rows larger than 128 MB.
- This request has been taken as a feature request and has been added to our engineering team's backlog for the next quarter to decide if we can allow the user to change GRPC_DEFAULT_OPTIONS
So as far as i understand it - you can't change the GRPC options when using databricks connect currently.