The solution using the option .option("cloudFiles.schemaEvolutionMode", "addNewColumns") is working as expected. With this configuration, when a new column appears in the source, the stream fails with an UnknownFieldException. However, we would like to capture any newly added columns using the schema drift (rescue column) mechanism. If this is not possible using the schema drift option, an alternative approach is required to print or log the names of the newly added columns so that business users can be notified.