Can't read multiline json from a unity catalog S3 external location
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-10-2022 03:12 PM
Hi,
Is there a know limitation in Unity Catalog workspaces with reading multiline json files from an S3 external location using pyspark? The file works perfectly fine if it's on dbfs, and reading json files in general works from that s3 location if they are single line, but when we try to set multiline option we get a scala NotImplementedError(). The DBR we tested this with is 10.4, and we have Databricks on AWS.
Thanks,
****
