11-16-2022 05:24 AM
Hello
I am trying to read this JSON file but didn't succeed
You can see the head of the file, JSON inside a list of lists. Any idea how to read this file?
11-21-2022 05:51 AM
Here is my solution, I am sure it can be optimized
import json
data=[]
with open(path_to_json_file, 'r') as f:
data.extend(json.load(f))
df = spark.createDataFrame(data[0], schema=schema)
✌️
11-17-2022 11:31 PM
Hi @Amine HADJ-YOUCEF , The data sources are limited, please refer: https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option
Also, https://docs.databricks.com/external-data/json.html
Please let us know if this helps.
11-18-2022 12:23 AM
Hi @Amine HADJ-YOUCEF, Please see these threads which have a solution to similar queries and LMK if that helps:-
11-21-2022 05:04 AM
Thank you for sharing,
these links do not address the exact problem I am facing
11-20-2022 11:11 PM
Hi @Amine HADJ-YOUCEF, We haven’t heard from you since the last response from @Debayan Mukherjee and me, and I was checking back to see if our suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
11-21-2022 05:06 AM
11-21-2022 05:51 AM
Here is my solution, I am sure it can be optimized
import json
data=[]
with open(path_to_json_file, 'r') as f:
data.extend(json.load(f))
df = spark.createDataFrame(data[0], schema=schema)
✌️
11-21-2022 11:15 AM
Hi @Amine HADJ-YOUCEF, Thank you for sharing the solution with the community!