szymon_dybczak
Esteemed Contributor III

Hi @ChristianRRL ,

No, autoloader is propriety to Databricks. It's not open sourced. Open source version of SDP uses spark structured streaming for incremental processing. 
Keep in mind that Auto Loader is basically just Spark streaming under the hood with additional features for event-driven ingestion (and some other things).
Schema evolution is property of Delta protocol which is open sourced. Also spark can natively infer schema for various sources, this is not something that is unique to auto loader.
InferSchema & Schema Enforcement in Spark | by Omkar Patil | Medium

View solution in original post