Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2024 02:29 AM
Hi @hpant would you consider testing the new VARIANT type for your JSON data? I appreciate it will require rewriting the next step in your pipeline, but should be more robust wrt errors.
Disclaimer: I haven't personally tested variant with Autoloader. It should speed up some reads, but I don't think it yet creates file statistics.
Alternatively, you can partially define a schema and put the remainder of the columns in a 'rescued' column, but it is a hassle splitting it out afterwards.