Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-19-2022 02:00 AM
Thank you @Debayan Mukherjee but I think I was misunderstood. Let me give you more details:
- I need to compare several Delta tables with different schema each with their analogue avro schemas
- I've managed to build a dataframe with the avro schemas using wholeTextFiles from spark RDD and I want to do something similar for the Delta schemas of those Delta parquet files
- Because those delta tables have different schemas I can't use the spark standard methods and I guess I need to do a loop in Scala through all those folders with parquet files and load each of them separately.
- But I wanted to know if there would be another method similar to wholeTextFiles fir text files.
Thank you,
Ovi