Just FYI, as of Jan 16th (the time I'm writing this answer), SDP and Delta Lake in their OSS versions don't work together yet.SDP is part of Apache Spark 4.1, but Delta Lake does not support it at the moment. It's coming. No idea when it's gonna be a...
Can you show some code to get the gist of what the code does? Are the parquet files accessed as a catalog table? Could it be that some other job makes changes to input tables?
That's exactly my words! I'd not be surprised if this were the author of DAB judging by the nickname (https://github.com/databricks/cli/commits?author=pietern) 
Why not use Substitutions and Custom variables that can be specified on command line using --var="<key>=<value>"?With all the features your databricks.yml would look as follows:variables: git_branch: default: maingit_source: git_url: https://git...
That's exactly my case!This is what I saw in `databricks bundle run my_job`:AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClientAnd...
