I am attempting to apply a function to a pyspark DataFrame and save the API response to a new column and then parse using `json_normalize`. This works fine in pandas, however, I run into an exception with `pyspark`. 

 import pyspark.pandas as ps
 
  import pandas as pd
 
  import requests
 
 
 
 
 
  def get_vals(row):
 
   
 
    # make api call
 
    return row['A'] * row['B']
 
   
 
 
 
  # Create a pandas DataFrame
 
  pdf = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
 
 
 
 
 
  # apply function - get api responses
 
  pdf['api_response'] = pdf.apply(lambda row: get_vals(row), axis=1)
 
  pdf.sample(5)
 
 
 
  # Unpack JSON API Response
 
  try:
 
    dff = pd.json_normalize(pdf['api_response'].str['location'])
 
  except TypeError as e:
 
    print(f"Error: {e}")
 
    print(f"Problematic data: {data['data']}")
 
 
  # To pySpark DataFrame
 
  psdf = ps.DataFrame(df)
 
  psdf.head(5)

Expected output is a `json` normalized DataFrame. When I attempt to apply the function over a `pyspark` DataFrame, it throws an exception: 

psdf['api_response'] = psdf.apply(lambda row: get_vals(row), axis=1)
 
    
 
 ---------------------------------------------------------------------------
 
  TypeError                 Traceback (most recent call last)
 
  File <command-4372401754138893>:2
 
     1 
 
  ----> 2 psdf['api_response'] = psdf.apply(lambda row: get_vals(row), axis=1)
 
   
 
  TypeError: Nested StructType not supported in conversion from Arrow: struct<data: struct<geographies: