This is helpful. Follow-up question, when setting up Databricks pipelines (previously DLT Pipelines), does it require that Autoloader is used, or can we set it to use spark structured streaming? Mainly asking to see how much vendor lock-in concerns may be eased if we can use SDP without having to use Autoloader if this is a path we want to consider.