- 3140 Views
- 4 replies
- 1 kudos
sample
Help parsing the JSON using Spark SQL or python. Sample json attached.
- 3140 Views
- 4 replies
- 1 kudos
Latest Reply
Forum Posts
- 7961 Views
- 5 replies
- 2 kudos
Split dataFrame into 1MB chunks and create a single json array with each row in chunk being an array element
Hi, I have a dataFrame that I've been able to convert into a struct with each row being a JSON object.I want the ability to split the data frame into 1MB chunks. Once I have the chunks, I would like to add all rows in each respective chunk into a sin...
- 7961 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
@Tamoor Mirza :You can use the to_json method of a DataFrame to convert each chunk to a JSON string, and then append those JSON strings to a list. Here is an example code snippet that splits a DataFrame into 1MB chunks and creates a list of JSON arr...
- 2 kudos
- 2719 Views
- 1 replies
- 3 kudos
How to extract JSON object from a pyspark data frame. I was able to extract data from another column which in array format using "Explode" function, but Explode is not working for Object type. Its returning with type mismatch error.
I have tried below code to extract data which in Array:df2 = df_deidentifieddocuments_tst.select(F.explode('annotationId').alias('annotationId')).select('annotationId.$oid')It was working fine.. but,its not working for JSON object type. Below is colu...
- 2719 Views
- 1 replies
- 3 kudos
Latest Reply
- 3 kudos
Did you try extracting that column data using from_json function ?
- 3 kudos
- 38454 Views
- 24 replies
- 4 kudos
Resolved! Extracting data from a multi-layered JSON object
I have a table in databricks called owner_final_delta with a column called contacts that holds data with this structure:array<struct<address:struct<apartment:string,city:string,house:string,poBox:string,sources:array<string>,state:string,street:strin...
- 38454 Views
- 24 replies
- 4 kudos
Latest Reply
- 4 kudos
Have you tried to use the explode function for that column with the array?df.select(explode(df.emailId).alias("email")).show()----------Also, if you are a SQL lover, you can instead use the Databricks syntax for querying a JSON seen here.
- 4 kudos