topic Re: How can we get AutoLoader to detect a file footer? in Data Engineering

How can we get AutoLoader to detect a file footer?

BF7 — Fri, 25 Apr 2025 13:58:18 GMT

We are dealing with CSVs that have footers in them. When we have an empty file, the presence of this footer seems to impair the schema inferencing of AutoLoader, because of the footer.

I know where is a header = true parameter, but I don't see a footer parameter in documentation.

Has anyone found a good way to deal with the presence of a FOOTER in CSV files being detected by AutoLoader?

Re: How can we get AutoLoader to detect a file footer?

Louis_Frolio — Fri, 25 Apr 2025 15:33:24 GMT

To be clear, when you say footer are you referring to the last row of the tuple? e.g. Header = row 1, Footer = row_last.

Re: How can we get AutoLoader to detect a file footer?

BF7 — Fri, 25 Apr 2025 16:09:43 GMT

If it were to be correctly read from a CSV into a dataframe in which the rows are tuples then yes, the footer would be the last row.

Re: How can we get AutoLoader to detect a file footer?

Louis_Frolio — Fri, 25 Apr 2025 16:18:22 GMT

Then no, there is no option as part of the Spark API to handle that. You would have to do some custom coding. Hope this help, Louis.