- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2023 01:49 AM - edited 09-04-2023 01:55 AM
Thanks so much for responding!
I dove into the metastore and saw that because of my partitioning (?), directories are created using actual values from the table. On checking the source data, I see that indeed one of the values ends with a period, so adding a sanitisation step to remove punctuation (a good idea in any case) should solve it. Obvious in retrospect to check this.
A bit of an awkward situation to deal with for the engine: I guess it's not trivial to automatically sanitise during the directory creation process as it would mess with lookups if the directory name doesn't match the table value.
Because partition column shouldn't generally be random long strings (containing punctuation) i guess it's not usually much of a problem.
Because this is a limitation of the engine it would be good if the exception and solution was mentioned in the documentation, though.
Thanks again!