Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Wednesday
Hi @Malthe ,
read_files accepts format-specific options, but the Avro option table has an important scope column. For mode, that scope is only from_avro, not Avro file reads. Spark documents it as:
"mode … allows you to specify parse mode for function from_avro"
As a workaround, if your blobs are raw Avro-encoded records/messages, rather than standard Avro Object Container Files, read them as binary and use from_avro, where mode = PERMISSIVE really is supported:
SELECT
from_avro(
content,
'<avro schema>',
map('mode', 'PERMISSIVE')
) AS decoded
FROM read_files(
's3://bucket/path',
format => 'binaryFile'
);