Table does not support batch scan.

[harsh120]

After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error:

[UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported:
Table `app_dev`.`vw_period` does not support batch scan. Please check the
current catalog and namespace to make sure the qualified table name is
expected, and also check the catalog implementation which is configured
by "spark.sql.catalog". SQLSTATE: 0A000

vw_period is a regular Unity Catalog view, and the table under it is a plain external Delta table, not a streaming table, materialized view or pipeline output. The same views, data and code ran without problems on 16.4.

Table and view Defination

-- external Delta table, about 20 rows in total
CREATE TABLE acme_prod.app_dev.calendar_entries (
  entry_id BIGINT, period_key BIGINT, period_label STRING,
  period_start DATE, period_end DATE, period_tz STRING
  -- other columns omitted
) USING DELTA;

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_period AS
SELECT DISTINCT period_key, period_label, period_start, period_end, period_tz
FROM acme_prod.app_dev.calendar_entries;

-- external Delta table, partitioned by activity_type; the real one has about 170 columns
CREATE TABLE acme_prod.app_dev.activity_events (
  activity_id STRING, actor_key BIGINT, activity_type STRING,
  platform_label STRING, session_id STRING,
  occurred_at TIMESTAMP, is_automated BOOLEAN
  -- other columns omitted
) USING DELTA PARTITIONED BY (activity_type);

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_activity_events AS
SELECT e.*, COALESCE(p.period_key, CAST(-1 AS BIGINT)) AS period_key
FROM acme_prod.app_dev.activity_events e
LEFT JOIN acme_prod.app_dev.vw_period p
  ON CAST(e.occurred_at AS DATE) BETWEEN p.period_start AND p.period_end
WHERE e.activity_type <> 'heartbeat';

What works and what fails

The first read works on DBR 18, and isEmpty() and rdd.isEmpty() both return normally on it:

src=spark.sql("""
  SELECT * FROM acme_prod.app_dev.vw_activity_events
  WHERE activity_type IN ('a', 'b')
    AND occurred_at >  TIMESTAMP '2026-10-13 09:02:00'
    AND occurred_at <= TIMESTAMP '2026-10-19 12:46:00'
    AND COALESCE(is_automated, false) = false
""")
src.createOrReplaceTempView("events_incr")

The next step is where it fails:

out = spark.sql("""
  SELECT CONCAT('X_', activity_id) AS event_key, actor_key,
         UPPER(activity_type) AS activity_type, CAST(occurred_at AS DATE) AS event_date,
         period_key,
         MAP('platform_label', platform_label, 'session_id', session_id) AS props
  FROM events_incr
""").filter("actor_key IS NOT NULL")

out.isEmpty()   # raises the error above

The plan for the first query is below. Under the outer view, vw_period is not expanded into its SELECT DISTINCT. It shows up as RelationV2 and is read by RemoteSparkConnectScan, while the outer view is expanded normally.

== Parsed Logical Plan ==
'Project [*]
+- 'Filter ((activity_type IN (...) AND (occurred_at > 2026-10-13 09:02:00.404))
            AND ((occurred_at <= 2026-10-19 12:46:43.923587)
                 AND (COALESCE(is_automated, false) = false)))
   +- 'UnresolvedRelation [app_dev, vw_activity_events], [], false

== Analyzed Logical Plan ==
Project [...]
+- Filter ((activity_type IN (...) AND (occurred_at > ...)) AND ((occurred_at <= ...) AND (coalesce(is_automated, false) = false)))
   +- SubqueryAlias acme_prod.app_dev.vw_activity_events
      +- View (acme_prod.app_dev.vw_activity_events, [...])
         +- Project [...]
            +- Filter NOT (activity_type = heartbeat)
               +- Join LeftOuter, between(cast(occurred_at as date), period_start, period_end)
                  :- SubqueryAlias acme_prod.app_dev.activity_events
                  :  +- Relation acme_prod.app_dev.activity_events[...] parquet
                  +- SubqueryAlias p
                     +- SubqueryAlias acme_prod.app_dev.vw_period
                        +- RelationV2[period_key, period_label, period_start, period_end, period_tz] acme_prod.app_dev.vw_period

== Optimized Logical Plan ==
Project [...]
+- Join LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end))
   :- Project [...]
   :  +- Filter (activity_type INSET (...) AND occurred_at > ... AND occurred_at <= ... AND NOT coalesce(is_automated, false))
   :     +- Relation acme_prod.app_dev.activity_events[...] parquet
   +- RelationV2[period_key, period_start, period_end] acme_prod.app_dev.vw_period

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end)), true
   :- FileScan parquet acme_prod.app_dev.activity_events[...]
   :     Batched: false
   :     Location: PreparedDeltaFileIndex(1 paths)[s3://example-bucket/dev/tenant/master-data/activity_events]
   :     PartitionFilters: [isnotnull(activity_type), NOT (activity_type = heartbeat), activity_type INSET (...)]
   :     PushedFilters: [IsNotNull(occurred_at), GreaterThan(occurred_at, ...), LessThanOrEqual(occurred_at, ...)]
   +- Exchange SinglePartition, EXECUTOR_BROADCAST
      +- Project [period_key, period_start, period_end]
         +- RemoteSparkConnectScan acme_prod.app_dev.vw_period[period_key, period_start, period_end]
              PushedPredicates: [period_start IS NOT NULL, period_end IS NOT NULL]
              Runtime row count reported for this side: 5
              Size: 160 bytes

What I tried

• Setting spark.sql.ansi.enabled to false made no difference.
• Recreating both views, and using fully qualified names, made no difference. DESCRIBE EXTENDED on both views shows the correct catalog and namespace.
• Calling localCheckpoint(eager=True) on the first DataFrame, before the second step, lets the whole job run on DBR 18. That is our workaround for now.
• A second job joins another Delta table to vw_activity_events and adds a transform() over an array column in the same query. It fails the same way. Splitting the join from the transform() with a checkpoint avoids it. I haven’t isolated which expression triggers it.

Is this a known issue in DBR 18? Why is vw_period planned as RelationV2 and read through RemoteSparkConnectScan instead of being expanded, and what makes the second query’s scan fail when the first query’s succeeds? Is there a setting or recommended pattern to use until a fix is out?

[data_pulse]

@harsh120 

I tried reproducing this with a similar setup using nested permanent Unity Catalog views, including SELECT DISTINCT, a date-range LEFT JOIN, a temporary view and the subsequent MAP()/filter transformations with isEmpty().

Both steps passed on Databricks Serverless (Spark 4.2.0), including with ANSI enabled. The execution plans expanded the UC views into the underlying Delta tables, using PhotonScan and PhotonBroadcastNestedLoopJoin.

One notable difference from your DBR 18 plan is RemoteSparkConnectScan / RelationV2 for vw_period, rather than expansion into the underlying Delta table.

Databricks documents RemoteSparkConnectScan in the context of fine-grained access control on Dedicated compute.

Since localCheckpoint(eager=True) works, I suspect the issue may involve how DBR 18 replans the nested view through this remote scan during Step 2, rather than the MAP() expression or ANSI mode itself.

Few questions to understand further:

1. Is this running on Dedicated compute, and does the job identity have SELECT on the underlying tables or only on the UC views?

2. If possible, could you share the Step 2 explain() plan both with and without the checkpoint? That would confirm whether RemoteSparkConnectScan disappears after materialisation.