09-11-2022 12:28 AM
Like @Joseph Kambourakis said, there are plenty of ways to do this. You can write pure Python or SQL. For me, it's easier to write SQL so I would first load this data into a Delta table and then write pure SQL.
