Databricks Community

rusty9876543 · 04-12-2023

Hi, I have a dataFrame that I've been able to convert into a struct with each row being a JSON object.I want the ability to split the data frame into 1MB chunks. Once I have the chunks, I would like to add all rows in each respective chunk into a sin...

rusty9876543 · 04-13-2023

Thanks thats very helpful! As far as "traversing across each partition and collect the rows individually. you can then add up these rows for each partition" would this be a simple for loop or is there a recommended/optimized way of traversing & colle...

rusty9876543 · 04-13-2023

Thanks thats very helpful! As far as "traversing across each partition and collect the rows individually. you can then add up these rows for each partition" would this be a simple for loop or is there a recommended/optimized way of traversing & colle...

Databricks Community

User Stats

User Activity

Split dataFrame into 1MB chunks and create a single json array with each row in chunk being an array element

Re: Split dataFrame into 1MB chunks and create a single json array with each row in chunk being an array element

Re: Split dataFrame into 1MB chunks and create a single json array with each row in chunk being an array element