- 4648 Views
- 3 replies
- 2 kudos
How to efficiently process a 100GiB JSON nested file and store it in Delta?
Hi, I'm a fairly new user and I am using Azure Databricks to process a ~1000GiB JSON nested file containing insurance policy data. I uploaded the JSON file to Azure Data Lake Gen2 storage and read the JSON file into a dataframe.df=spark.read.option("...
- 4648 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi Sameer, please refer to following documents on how to work with nested json:https://docs.databricks.com/optimizations/semi-structured.htmlhttps://learn.microsoft.com/en-us/azure/databricks/kb/_static/notebooks/scala/nested-json-to-dataframe.html
- 2 kudos
- 1916 Views
- 0 replies
- 1 kudos
How to dynamically convert Spark DataFrame to Nested json using Spark Scala
I want to convert the DataFrame to nested json. Sourse Data:-DataFrame have data value like :- As image 2 Expected Output:-I have to convert DataFrame value to Nested Json like : -As image 1Appreciate your help !
- 1916 Views
- 0 replies
- 1 kudos
- 13135 Views
- 1 replies
- 3 kudos
Resolved! How to Efficiently Read Nested JSON in PySpark?
I am having trouble efficiently reading & parsing in a large number of stream files in Pyspark! Context Here is the schema of the stream file that I am reading in JSON. Blank spaces are edits for confidentiality purposes. root |-- location_info: ar...
- 13135 Views
- 1 replies
- 3 kudos
- 3 kudos
I'm interested in seeing what others have come up with. Currently I'm using Json. normalize() then taking any additional nested statements and using a loop to pull them out -> re-combine them.
- 3 kudos