I'm trying to save the returned json data from a requests API call to a Delta Table. I get a ParseError when I INSERT the response object which is in json format. The error shows the json data and a marker that states a ' or } or ) is missing. I validated the json data. Why is the INSERT process trying to parse the json data? Is there another way to INSERT the data?
json_data = response.text
query = f"INSERT INTO database.table VALUES ('{json_data}')"
spark.sql(query)
