yit
Databricks Partner

 @szymon_dybczak Yes, I’m aware of that behavior and actually expect it. That’s why I’m handling it explicitly in the exception clause: 

 if "UnknownFieldException" in error_msg and "automatic retry: true" in error_msg: print("Schema evolution detected. Retrying with updated schema...")

This works perfectly - it allows the loop to continue and rerun the autoloader with the updated schema. (It actually enters many times in the exception because of this, and continues with the updated schema, so it works fine.)

What’s strange is that the upper error message ("Some streams terminated before this command could finish!") appears after the autoloader has finished running. The logs even show ‘Streaming job ended.’, but the notebook cell still displays an ERROR. This cell contains only the autoloader code.