topic Re: How to load a json file in pyspark with colon character in file name in Data Engineering

How to load a json file in pyspark with colon character in file name

laus — Thu, 21 Apr 2022 13:55:25 GMT

Hi,

I'm trying to load this json file which contains the colon character in its name: file_name.2022-03-05_11:30:00.json but I get the error in screenshot below saying that there is a relative path in an absolute url -

Any idea how to read this file?

file_name = file_name.2022-03-05_11:30:00.json

Thanks for your help in advance!

Re: How to load a json file in pyspark with colon character in file name

User16741082858 — Thu, 21 Apr 2022 14:54:34 GMT

Hi @Laura Blancarte . Have you tried saving the file path to a list?

Re: How to load a json file in pyspark with colon character in file name

laus — Thu, 21 Apr 2022 15:31:40 GMT

Hi @Pearl Ubaru ,

Thanks for your reply.

Yes, I have a list of path files through which I loop and try to load the file at the current index but I get the same error about there being a relative path. Am I missing something?

Ty!

Re: How to load a json file in pyspark with colon character in file name

User16741082858 — Thu, 21 Apr 2022 16:00:04 GMT

Is it okay if I see a bit more of your code? Preferably the list and loop?

Re: How to load a json file in pyspark with colon character in file name

laus — Thu, 21 Apr 2022 16:37:50 GMT

@Pearl Ubaru , Sure, here it is:

Re: How to load a json file in pyspark with colon character in file name

User16741082858 — Tue, 26 Apr 2022 20:21:26 GMT

Hi @Laura Blancarte , I apologize for the late response... Try replacing the colons in the file name with %3A. This will work by replacing the colons with URL encoded format.

Re: How to load a json file in pyspark with colon character in file name

Noopur_Nigam — Tue, 02 Aug 2022 04:50:31 GMT

Hi @Laura Blancarte I hope that @Pearl Ubaru's answer would have helped you in resolving your issue.Please let us know if you need more help on this.