I ended up using the Databricks SDK pipelines update before each run (a hack), to update the configuration. 

I think the DLT side of Databricks is its own world; the jobs and repository configuration works differently than other features, and it has only just been fully integrated with unity catalog (2025 DLT Update: Intelligent, fully governed data pipelines | Databricks Blog). It would be nice for some consistency...