abohlin
New Contributor II

Came across this thread as I was facing the same exact issue as @malterializedvw and I want to comment my fix in case anyone else tears their hair out on this problem.

In my databricks.yml file I put in a gold_catalog and a silver_catalog variable and then passed those to the sql_task in a job.yml similar to how Pat showed.  The only difference is I use ${var.variable_name} call in the job.yml instead of doing a hard-coded redefine.

To solve the Exactly one CREATE [ LIVE TABLE | MATERIALIZED VIEW | STREAMING TABLE ] statement expected, but 0 found issue, I had to use the USE CATALOG IDENTIFIER({{gold_catalog}}); and then you can put the create statement without using a variable or running into the quote injection issue that {{var}} calls cause.

I ran into this [UNBOUND_SQL_PARAMETER] error when I was using FROM IDENTIFIER(:silver_catalog || '.schema_example.table_example') and got around that by using {{silver_catalog}} instead of :silver_catalog.

Just seems odd that some ways of calling variables seem to persist in MV's while others don't. I do wonder why {{var}} is fine, but :var causes it to choke. I would expect them both to resolve the same way, but it might be that {{var}} gets subbed out before the MV SQL gets stored, but :var doesn't so databricks is trying to store SQL with a variable that isn't defined anymore.