AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=`
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-16-2026 09:54 PM
Hi all,
We are experiencing an AnalysisException when migrating our Databricks cluster runtime from 13.3 LTS to 16.4 LTS (Scala 2.13), and would appreciate your help in identifying the root cause.
**Error:**
```
AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=` on search path
[`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].
SQLSTATE: 42883
```
**Where it fails:**
The exception is thrown inside a Scala `object` class when calling `.where()` on a DataFrame — specifically at:
```
org.apache.spark.sql.Dataset.filter(Dataset.scala:1993)
org.apache.spark.sql.Dataset.where(Dataset.scala:2021)
```
The filter uses the Column DSL (e.g. `col("job_id") === lit(jobId)`), not raw SQL strings.
**What we have tried:**
- Replacing `.where("enabled = 1")` string predicates with Column DSL equivalents (`===`, `=!=`) — same error
- Confirming `SHOW FUNCTIONS LIKE '='` returns the function as expected
- Verifying `rawDF.printSchema()` succeeds — the DataFrame itself is healthy
**Questions:**
1. Is this a known regression or breaking change in the HybridAnalyzer between DBR 13.3 LTS and 16.4 LTS?
2. Is there a recommended way to ensure built-in operators are resolved correctly when the active catalog is not `spark_catalog`?
3. Is setting `USE CATALOG spark_catalog` at session start the intended workaround, or is there a runtime/cluster config that should handle this?
Thank you
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2026 09:50 AM
You can add a cluster level Spark configuration to set the default catalog at startup avoiding per session configurations.
spark.databricks.sql.initial.catalog.name spark_catalogYou can use spark.sql.path.enabled = true and explicitly set PATH to include system builtin
spark.conf.set("spark.sql.path.enabled", "true")
spark.sql("SET PATH = system.builtin, system.session, spark_catalog.default")
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2026 03:09 PM
It doesn't sound like the === operator itself is the issue, since it should compile into a normal Catalyst expression rather than a SQL function call. The fact that Spark is trying to resolve = as a routine makes me suspect a catalog or analyzer behavior change in DBR 16.4. check whether a custom catalog, Unity Catalog setting, or Spark extension is intercepting analysis before assuming it's a regression. If you can share a minimal reproducible example and your catalog/session configuration, it'll be much easier to narrow down.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-04-2026 07:55 AM
Having a similar issue when moving from DBR 13 to DBR 16.4 in a multi-task Job. The first task works ok but the 2nd with identical code but different params fails. For me it fails with Cannot resolve function `and` (I assume from the .between(from_date, to_date)).
Anyone solved it? or knows of a workaround?