topic Can't read large multiline json, in Data Engineering

Can't read large multiline json,

espenol — Thu, 22 Dec 2022 13:31:37 GMT

Hey! So I'm struggling to read a multiline json. Some details:

It's gzipped from the API I get it from
just a single file in the folder currrently
stored in ADLS Gen2 storage.
95 MB zipped, approximately 1.2 GB unzipped

I can read it just fine using the text read:

But, if I try to read it normally as json without multiline option, I get corrupted string after some time reading(perhaps as expected)

But, if I instead use multiline option then I immediately get an error:

Can anyone give me some pointers towards what is wrong? If I uncomment the last line I get the same error:

IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: enheter_full_2022-12-22T13:07:10.4988008Z_b744d2f4-5eb5-41a0-a546-e0514c7db325.json.gz

Does anyone know how to fix this? All the googling I've done suggests this is a problem with my path starting with "/", but my path starts with abfss:...

Re: Can't read large multiline json,

-werners- — Thu, 22 Dec 2022 13:41:33 GMT

it seems to be your url, and the last part to be more precise.

what happens if you pass the complete path?

Re: Can't read large multiline json,

daniel_sahal — Thu, 22 Dec 2022 13:49:51 GMT

That's a pretty old issue with having ':' sign in a file name.

As of now there's no perfect workaround other than simply renaming the file or moving file names into a list (needs to be tested).

You can read more here:

https://stackoverflow.com/questions/48909921/struggling-with-colon-in-file-names

Re: Can't read large multiline json,

espenol — Thu, 22 Dec 2022 14:00:15 GMT

Thanks a lot for the help! Removing colon fixed it. Now I need to fix the Data Factory instance that writes to my storage container. Hope it's easy, Data Factory is such a hassle.

Re: Can't read large multiline json,

jose_gonzalez — Tue, 27 Dec 2022 23:13:31 GMT

Please mark any if the given responses as best. Thank you in advance.