What's the best way to get from Python dict > JSON > PySpark and apply as a mapping to a dataframe?

397973
New Contributor III

‎04-02-2025 12:13 PM

I'm migrating code from Python Linux to Databricks PySpark. I have many mappings like this: 

{
    "main": {
    "honda": 1.0,
    "toyota": 2.9,
    "BMW": 5.77,
    "Fiat": 4.5,
    },
}

I exported using json.dump, saved to s3 and was able to import with spark.read.json, but that puts it into a dataframe with nested objects, like this. I can access the objects with select but don't know how to apply it as a mapping to another dataframe. 

 

397973_0-1743620626332.png

Another approach is to save as JSON Lines and read in as a dictionary? From what I understand PySpark prefers to have JSON lines rather than usual JSON, like this: 

{"main":{"honda":1,"toyota":2,"BMW":5,"Fiat":4}}

Ok so I manually created the dict and mapping. Given a dictionary car_map with the values, and a dataframe df:

 

map_col = F.create_map([F.lit(x) for i in car_map.items() for x in i])
df.withColumn('x', map_col[F.col('car')]).display()
 
So then how do I create the JSON lines in Python? It would be a list of dicts but I need to label them. That would make it a JSON rather than JSON lines? 
 
What approach do you think is better, or is there another? 
0 Kudos