How can we get AutoLoader to detect a file footer?

BF7
Contributor

‎04-25-2025 06:58 AM

We are dealing with CSVs that have footers in them. When we have an empty file, the presence of this footer seems to impair the schema inferencing of AutoLoader, because of the footer.

I know where is a header = true parameter, but I don't see a footer parameter in documentation. 

Has anyone found a good way to deal with the presence of a FOOTER in CSV files being detected by AutoLoader?

0 Kudos

Louis_Frolio
Databricks Employee
Databricks Employee

‎04-25-2025 08:33 AM

To be clear, when you say footer are you referring to the last row of the tuple?  e.g. Header = row 1, Footer = row_last.

 

0 Kudos

BF7
Contributor
In response to Louis_Frolio

‎04-25-2025 09:09 AM

If it were to be correctly read from a CSV into a dataframe in which the rows are tuples then yes, the footer would be the last row.

Louis_Frolio
Databricks Employee
Databricks Employee
In response to BF7

‎04-25-2025 09:18 AM

Then no, there is no option as part of the Spark API to handle that. You would have to do some custom coding. Hope this help, Louis.

View solution in original post