balajij8
Contributor III

Hi Vikas, 

You may share the code directly in a message. 

You can follow below

  • Enable case-sensitive mode spark.conf.set("spark.sql.caseSensitive", "true") & validate the code
  • Rename Fields immediately after parsing & validate the code.
parsed_trade_df = (
    trade_raw_df
    .select(from_json(col("value").cast("string"), trade_schema).alias("json"))
)​

changed_df = parsed_trade_df
    .select(col("json.t").alias("trade_id"),col("json.T").alias("trade_time_ms"), col("json.e").alias("event_type"))
)

 

View solution in original post