I started databricks native metadata driven ingestion framework: https://github.com/nitindatta/metabricks
