AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=`

[phanvy5403]

Hi all,

We are experiencing an AnalysisException when migrating our Databricks cluster runtime from 13.3 LTS to 16.4 LTS (Scala 2.13), and would appreciate your help in identifying the root cause.

**Error:**
```
AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=` on search path
[`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].
SQLSTATE: 42883
```

**Where it fails:**
The exception is thrown inside a Scala `object` class when calling `.where()` on a DataFrame — specifically at:
```
org.apache.spark.sql.Dataset.filter(Dataset.scala:1993)
org.apache.spark.sql.Dataset.where(Dataset.scala:2021)
```
The filter uses the Column DSL (e.g. `col("job_id") === lit(jobId)`), not raw SQL strings.

**What we have tried:**
- Replacing `.where("enabled = 1")` string predicates with Column DSL equivalents (`===`, `=!=`) — same error
- Confirming `SHOW FUNCTIONS LIKE '='` returns the function as expected
- Verifying `rawDF.printSchema()` succeeds — the DataFrame itself is healthy

**Questions:**
1. Is this a known regression or breaking change in the HybridAnalyzer between DBR 13.3 LTS and 16.4 LTS?
2. Is there a recommended way to ensure built-in operators are resolved correctly when the active catalog is not `spark_catalog`?
3. Is setting `USE CATALOG spark_catalog` at session start the intended workaround, or is there a runtime/cluster config that should handle this?

Thank you

[balajij8]

You can add a cluster level Spark configuration to set the default catalog at startup avoiding per session configurations.

spark.databricks.sql.initial.catalog.name spark_catalog

You can use spark.sql.path.enabled = true and explicitly set PATH to include system builtin

spark.conf.set("spark.sql.path.enabled", "true")
spark.sql("SET PATH = system.builtin, system.session, spark_catalog.default")

[henry_collins]

It doesn't sound like the === operator itself is the issue, since it should compile into a normal Catalyst expression rather than a SQL function call. The fact that Spark is trying to resolve = as a routine makes me suspect a catalog or analyzer behavior change in DBR 16.4. check whether a custom catalog, Unity Catalog setting, or Spark extension is intercepting analysis before assuming it's a regression. If you can share a minimal reproducible example and your catalog/session configuration, it'll be much easier to narrow down.