Converting 400mn+ rows into JSON, in my opinion is not a good solution, as it'll take a lot of space for no reason.

Anyway, so you've JSONlines in file but you want it to be JSON object only in the file. There's a simpler way to do this.

Let Spark write your data with 400mn+ records into 'x' number of JSON files.

Since databricks cells supports shell commands, you can run following script to convert JSONL to JSON files. Run it recursively or however you would like it.

Let's say your blob store location is mounted on dbfs in mnt directory.

%sh
cat /dbfs/mnt/<path to JSONlines input file> | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/,/g' | sed 's/n/,/' | sed 's/^/[/'| sed 's/$/]/' > /dbfs/mnt/<path to JSON output file>

The above command should convert your files in seconds.

Do share on how it goes with this approach.

Credit: Medium post.