Nowadays we already use the autoloader with checkpoint location, but I still wanted to know if it is possible to read only the last updated file within a folder. I know it somewhat loses the purpose of checkpoint locatio
Another question is it possible to obtain any information if the autoloader returns an empty dataframe without having to run a count on the dataframe? Some parameter of how many files or mb were read.