Louis_Frolio
Databricks Employee
Databricks Employee

Hello @harsh120, I took a look at both internal and external documentation and here is what I found.

Building on @data_pulse and @Jen_Evangelista: RemoteSparkConnectScan is the key clue, and this doesn't look like MAP(), isEmpty(), or ANSI mode being unsupported on their own.

On Dedicated compute, Databricks delegates part of a query to serverless compute when fine-grained access control is involved, and that delegation shows up in the plan as RemoteSparkConnectScan. The documented triggers are dynamic views, tables with row filters or column masks, materialized views, streaming tables, and views over tables the querying user lacks SELECT on. None of those should apply to vw_period, since your identity owns calendar_entries and has SELECT on it directly. 16.4 inlined the view; 18 and 19 route it remotely while expanding vw_activity_events normally. I couldn't find a release note or doc that explains why a nested view would be treated differently from the outer one.

The error adds a second clue. "Does not support batch scan" is Spark's UNSUPPORTED_FEATURE.TABLE_OPERATION message for a DataSource V2 table object that doesn't implement the readable interface. Your first query got a readable remote table; the second, resolved through the events_incr temp view, got one it couldn't read. Same view, different outcome depending on how the plan is reached. That's a planner inconsistency on the remote-filtering path, which lines up with Jen's point about treating the checkpoint as a plan boundary rather than evidence about any specific expression. I can't confirm a published fix for this case, so I'd open a support ticket with the two explain plans you already have, the exact runtime build string, and the fact that it reproduces on 18 and 19 but not 16.4. One housekeeping item first: DBR 18 now ships updates as dated entries that clusters pick up on restart, so make sure the cluster has been restarted recently and record the build you're on.

Checks that either confirm the mismatch or expose a legitimate trigger:

  • EXPLAIN SELECT * FROM acme_prod.app_dev.vw_period on its own on the DBR 18 cluster. If a standalone read shows RemoteSparkConnectScan, the routing decision is about the view itself, not the nesting.
  • SHOW CREATE TABLE on the view, plus acme_prod.information_schema.row_filters and column_masks for calendar_entries, and a look for any ABAC policy at the catalog or schema level that targets a tag on one of its columns. ABAC attaches above the table, so a table can pick up a mask without anyone touching its definition.
  • system.billing.usage filtered to billing_origin_product = 'FINE_GRAINED_ACCESS_CONTROL' for the job's run-as identity. If the cluster is paying serverless DBUs to read a 20-row view you own, that's a cost problem on top of the failure and good evidence for the ticket.

On workarounds, you have cleaner options than localCheckpoint. Jen's caution stands: local checkpoints live in executor storage and don't survive executor loss. In rough order of how clean they are:

  • Remove the nesting. vw_period is a SELECT DISTINCT projection, so fold it into vw_activity_events as a subquery over calendar_entries. Base tables you have SELECT on don't take the remote path.
  • Skip the temp view hop and keep chaining on src (src.selectExpr(...).filter(...)). The failure appears when the plan goes through the temp view a second time, so this is a cheap test, though I haven't verified it.
  • If you need a materialization boundary, write the slice to a Delta staging table, or use checkpoint(eager=True) with sc.setCheckpointDir pointed at a Unity Catalog volume, which is supported on Dedicated compute and restart-safe.
  • Run the minimal repro on Standard access mode or serverless. Standard enforces fine-grained access natively with no remote scan, and data_pulse's repro passed on serverless. If it passes there and fails on Dedicated, that isolates the issue to the remote-filtering path.

Downgrading to 16.4 is a reasonable short-term move; it's LTS through May 2028. I'd still file the ticket, since engineering is the only group that can explain the routing change.

References:

Regards,
Louis