Ovi
New Contributor III

Thank you @Debayan Mukherjee​ but I think I was misunderstood. Let me give you more details:

  • I need to compare several Delta tables with different schema each with their analogue avro schemas
  • I've managed to build a dataframe with the avro schemas using wholeTextFiles from spark RDD and I want to do something similar for the Delta schemas of those Delta parquet files
  • Because those delta tables have different schemas I can't use the spark standard methods and I guess I need to do a loop in Scala through all those folders with parquet files and load each of them separately.
  • But I wanted to know if there would be another method similar to wholeTextFiles fir text files.

Thank you,

Ovi