- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-20-2022 12:46 AM
I struggle to understand the question, so please correct me here:
If I understand correctly you have an excel filled with sql expressions (or field mappings source-sink) and want to use the content of that excel to insert it into spark code?
Technically I think it is possible, you could read the excel file into python or into a spark DF and extract the values (f.e. with the collect() function).
But is this really the way you want to go? Because basically you put your mapping logic into an excel file, which is opening the gates to hell IMO.
I would rather go for a selectExpr() expression. Like that the mappings reside into code, you can check it in into git, have versioning etc.