How can we get AutoLoader to detect a file footer?

[BF7]

We are dealing with CSVs that have footers in them. When we have an empty file, the presence of this footer seems to impair the schema inferencing of AutoLoader, because of the footer.

I know where is a header = true parameter, but I don't see a footer parameter in documentation. 

Has anyone found a good way to deal with the presence of a FOOTER in CSV files being detected by AutoLoader?

[BigRoux]

To be clear, when you say footer are you referring to the last row of the tuple?  e.g. Header = row 1, Footer = row_last.

[BF7]

If it were to be correctly read from a CSV into a dataframe in which the rows are tuples then yes, the footer would be the last row.

[BigRoux · Accepted solution]

Then no, there is no option as part of the Spark API to handle that. You would have to do some custom coding. Hope this help, Louis.