- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2024 10:58 AM
hi @filipniziol - thanks for replying. I realized it might actually be a different nested (child) column that has a period for the event_properties. (I'm showing it below) This is most likely the issue, but how can i keep the nested values as nested and not split out for the event_properties? I've believe what I could do is convert to a pandas dataframe becuase I'm able to load this as such but then i have to convert my Spark df to pandas df then convert back to spark df to load to a databricks table seems like alot.
|-- event_properties: struct (nullable = true)
| |-- Active.Elapsed Minutes: double (nullable = true)
| |-- Active.Frequency: long (nullable = true)
| |-- Active.Maximum: double (nullable = true)
| |-- Active.Minimum: double (nullable = true)
This is what event_properties column looks like in a pandas dataframe which is what I want it to look like for spark