topic Relative path in absolute URI when reading a folder with files containing ":" colons in filename in Data Engineering

Relative path in absolute URI when reading a folder with files containing ":" colons in filename

chanansh — Wed, 11 Jan 2023 17:42:02 GMT

I am trying to read a folder with partition files where each partition is date/hour/timestamp.csv where timestamp is the exact timestamp in ISO format, e.g. 09-2022-12-05T20:35:15.2786966Z

It seems like spark having issues with reading files with colons, which is quiet ridiculous.

The issue was raised in 2019 https://issues.apache.org/jira/browse/SPARK-28841 but it seems like the issue remains.

What is the solution, rather than now renaming zillions of file names, which in s3 requires copy...

I am using Spark 3.2.1

Re: Relative path in absolute URI when reading a folder with files containing ":" colons in filename

chanansh — Tue, 17 Jan 2023 09:40:22 GMT

I have renamed the files replacing : with - as the bug still exists

Re: Relative path in absolute URI when reading a folder with files containing ":" colons in filename

Debayan — Wed, 11 Jan 2023 22:04:17 GMT

The issue was reopened again https://issues.apache.org/jira/browse/HDFS-14762