lateral view explode in databricks - need help
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2024 10:33 PM
We are working on snowflake to databricks migration and we encountered the lateral flatten function of snowflake which we tried to convert to lateral view explode in databricks- but its output is a subset of lateral flatten
-----------------------
The output of this flatten contains a result containing 6 elements as shown on this page with (index,value) being major in some of the queries
In case of tables with multiple array columns - these index values are used to combine data stored in 2 different array columns by performing joins
e.g.
col1 <1,2,3>, col2 <abc,def,ghi>
indices of 1 and abc, 2 and def, 3 and ghi are same and they are related data values stored in 2 different columns in a table
--------------------------------------
However,in Databricks - its equivalent is LATERAL VIEW [OUTER] EXPLODE - which just returns the exploded values and not their indices.
this doesn't allow us to combine such data items as mentioned above
---------------------------
Any suggestions would be helpful
