Table does not support batch scan.

[harsh120]

After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error:

[UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported:
Table `app_dev`.`vw_period` does not support batch scan. Please check the
current catalog and namespace to make sure the qualified table name is
expected, and also check the catalog implementation which is configured
by "spark.sql.catalog". SQLSTATE: 0A000

vw_period is a regular Unity Catalog view, and the table under it is a plain external Delta table, not a streaming table, materialized view or pipeline output. The same views, data and code ran without problems on 16.4.

Table and view Defination

-- external Delta table, about 20 rows in total
CREATE TABLE acme_prod.app_dev.calendar_entries (
  entry_id BIGINT, period_key BIGINT, period_label STRING,
  period_start DATE, period_end DATE, period_tz STRING
  -- other columns omitted
) USING DELTA;

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_period AS
SELECT DISTINCT period_key, period_label, period_start, period_end, period_tz
FROM acme_prod.app_dev.calendar_entries;

-- external Delta table, partitioned by activity_type; the real one has about 170 columns
CREATE TABLE acme_prod.app_dev.activity_events (
  activity_id STRING, actor_key BIGINT, activity_type STRING,
  platform_label STRING, session_id STRING,
  occurred_at TIMESTAMP, is_automated BOOLEAN
  -- other columns omitted
) USING DELTA PARTITIONED BY (activity_type);

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_activity_events AS
SELECT e.*, COALESCE(p.period_key, CAST(-1 AS BIGINT)) AS period_key
FROM acme_prod.app_dev.activity_events e
LEFT JOIN acme_prod.app_dev.vw_period p
  ON CAST(e.occurred_at AS DATE) BETWEEN p.period_start AND p.period_end
WHERE e.activity_type <> 'heartbeat';

What works and what fails

The first read works on DBR 18, and isEmpty() and rdd.isEmpty() both return normally on it:

src=spark.sql("""
  SELECT * FROM acme_prod.app_dev.vw_activity_events
  WHERE activity_type IN ('a', 'b')
    AND occurred_at >  TIMESTAMP '2026-10-13 09:02:00'
    AND occurred_at <= TIMESTAMP '2026-10-19 12:46:00'
    AND COALESCE(is_automated, false) = false
""")
src.createOrReplaceTempView("events_incr")

The next step is where it fails:

out = spark.sql("""
  SELECT CONCAT('X_', activity_id) AS event_key, actor_key,
         UPPER(activity_type) AS activity_type, CAST(occurred_at AS DATE) AS event_date,
         period_key,
         MAP('platform_label', platform_label, 'session_id', session_id) AS props
  FROM events_incr
""").filter("actor_key IS NOT NULL")

out.isEmpty()   # raises the error above

The plan for the first query is below. Under the outer view, vw_period is not expanded into its SELECT DISTINCT. It shows up as RelationV2 and is read by RemoteSparkConnectScan, while the outer view is expanded normally.

== Parsed Logical Plan ==
'Project [*]
+- 'Filter ((activity_type IN (...) AND (occurred_at > 2026-10-13 09:02:00.404))
            AND ((occurred_at <= 2026-10-19 12:46:43.923587)
                 AND (COALESCE(is_automated, false) = false)))
   +- 'UnresolvedRelation [app_dev, vw_activity_events], [], false

== Analyzed Logical Plan ==
Project [...]
+- Filter ((activity_type IN (...) AND (occurred_at > ...)) AND ((occurred_at <= ...) AND (coalesce(is_automated, false) = false)))
   +- SubqueryAlias acme_prod.app_dev.vw_activity_events
      +- View (acme_prod.app_dev.vw_activity_events, [...])
         +- Project [...]
            +- Filter NOT (activity_type = heartbeat)
               +- Join LeftOuter, between(cast(occurred_at as date), period_start, period_end)
                  :- SubqueryAlias acme_prod.app_dev.activity_events
                  :  +- Relation acme_prod.app_dev.activity_events[...] parquet
                  +- SubqueryAlias p
                     +- SubqueryAlias acme_prod.app_dev.vw_period
                        +- RelationV2[period_key, period_label, period_start, period_end, period_tz] acme_prod.app_dev.vw_period

== Optimized Logical Plan ==
Project [...]
+- Join LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end))
   :- Project [...]
   :  +- Filter (activity_type INSET (...) AND occurred_at > ... AND occurred_at <= ... AND NOT coalesce(is_automated, false))
   :     +- Relation acme_prod.app_dev.activity_events[...] parquet
   +- RelationV2[period_key, period_start, period_end] acme_prod.app_dev.vw_period

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end)), true
   :- FileScan parquet acme_prod.app_dev.activity_events[...]
   :     Batched: false
   :     Location: PreparedDeltaFileIndex(1 paths)[s3://example-bucket/dev/tenant/master-data/activity_events]
   :     PartitionFilters: [isnotnull(activity_type), NOT (activity_type = heartbeat), activity_type INSET (...)]
   :     PushedFilters: [IsNotNull(occurred_at), GreaterThan(occurred_at, ...), LessThanOrEqual(occurred_at, ...)]
   +- Exchange SinglePartition, EXECUTOR_BROADCAST
      +- Project [period_key, period_start, period_end]
         +- RemoteSparkConnectScan acme_prod.app_dev.vw_period[period_key, period_start, period_end]
              PushedPredicates: [period_start IS NOT NULL, period_end IS NOT NULL]
              Runtime row count reported for this side: 5
              Size: 160 bytes

What I tried

• Setting spark.sql.ansi.enabled to false made no difference.
• Recreating both views, and using fully qualified names, made no difference. DESCRIBE EXTENDED on both views shows the correct catalog and namespace.
• Calling localCheckpoint(eager=True) on the first DataFrame, before the second step, lets the whole job run on DBR 18. That is our workaround for now.
• A second job joins another Delta table to vw_activity_events and adds a transform() over an array column in the same query. It fails the same way. Splitting the join from the transform() with a checkpoint avoids it. I haven’t isolated which expression triggers it.

Is this a known issue in DBR 18? Why is vw_period planned as RelationV2 and read through RemoteSparkConnectScan instead of being expanded, and what makes the second query’s scan fail when the first query’s succeeds? Is there a setting or recommended pattern to use until a fix is out?

[data_pulse]

@harsh120 

I tried reproducing this with a similar setup using nested permanent Unity Catalog views, including SELECT DISTINCT, a date-range LEFT JOIN, a temporary view and the subsequent MAP()/filter transformations with isEmpty().

Both steps passed on Databricks Serverless (Spark 4.2.0), including with ANSI enabled. The execution plans expanded the UC views into the underlying Delta tables, using PhotonScan and PhotonBroadcastNestedLoopJoin.

One notable difference from your DBR 18 plan is RemoteSparkConnectScan / RelationV2 for vw_period, rather than expansion into the underlying Delta table.

Databricks documents RemoteSparkConnectScan in the context of fine-grained access control on Dedicated compute.

Since localCheckpoint(eager=True) works, I suspect the issue may involve how DBR 18 replans the nested view through this remote scan during Step 2, rather than the MAP() expression or ANSI mode itself.

Few questions to understand further:

1. Is this running on Dedicated compute, and does the job identity have SELECT on the underlying tables or only on the UC views?

2. If possible, could you share the Step 2 explain() plan both with and without the checkpoint? That would confirm whether RemoteSparkConnectScan disappears after materialisation.

[harsh120]

1. Is this running on Dedicated compute, and does the job identity have SELECT on the underlying tables or only on the UC views?

Yes, it is Dedicated access mode on DBR 18 (Spark 4.1.0, also tested on DBR 19, spark 4.2). The identity running the query owns the views and the underlying Delta tables, and also has catalog-level privileges (SELECT and ALL PRIVILEGES) through a group. So it has direct SELECT on the base tables, not only on the views.

2. If possible, could you share the Step 2 explain() plan both with and without the checkpoint? That would confirm whether RemoteSparkConnectScan disappears after materialisation.

Both plans are for the Step 1 read (SELECT * from vw_activity_events), shortened to the first 8 columns of each list. Table and column names are changed.

Without the checkpoint:

== Parsed Logical Plan ==
'Project [*]
+- 'Filter (('activity_type IN (a, b) AND ('occurred_at > 2026-10-13 09:02:00.404)) AND (('occurred_at <= 2026-10-20 06:36:48.614052) AND ('COALESCE('is_automated, false) = false)))
+- 'UnresolvedRelation [app_dev, vw_activity_events], [], false

== Analyzed Logical Plan ==
load_date: date, year: int, month: int, day: int, app_id: string, page_view_id: string, actor_key: bigint, user_id: string, ... 161 more fields
Project [load_date#2396, year#2397, month#2398, day#2399, app_id#2400, page_view_id#2401, actor_key#2402L, user_id#2403, ... 161 more fields]
+- Filter ((activity_type#2526 IN (a, b) AND (occurred_at#2524 > 2026-10-13 09:02:00.404)) AND ((occurred_at#2524 <= 2026-10-20 06:36:48.614052) AND (coalesce(is_automated#2563, false) = false)))
+- SubqueryAlias acme_prod.app_dev.vw_activity_events
+- View (`acme_prod`.`app_dev`.`vw_activity_events`, [load_date#2396, year#2397, month#2398, day#2399, app_id#2400, page_view_id#2401, actor_key#2402L, user_id#2403, ... 161 more fields])
+- Project [cast(load_date#2756 as date) AS load_date#2396, cast(year#2757 as int) AS year#2397, cast(month#2758 as int) AS month#2398, cast(day#2759 as int) AS day#2399, cast(app_id#2760 as string) AS app_id#2400, cast(page_view_id#2761 as string) AS page_view_id#2401, cast(actor_key#2941L as bigint) AS actor_key#2402L, cast(user_id#2776 as string) AS user_id#2403, ... 161 more fields]
+- Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, actor_key#2941L, user_id#2776, ... 161 more fields]
+- Filter NOT (activity_type#2885 = heartbeat)
+- Join LeftOuter, between(cast(occurred_at#2883 as date), period_start#2954, period_end#2955)
:- SubqueryAlias e
: +- SubqueryAlias acme_prod.app_dev.activity_events
: +- Relation acme_prod.app_dev.activity_events[load_date#2756,year#2757,month#2758,day#2759,app_id#2760,page_view_id#2761,session_id#2762,domain_sessionidx#2763,... 183 more fields] parquet
+- SubqueryAlias p
+- SubqueryAlias acme_prod.app_dev.vw_period
+- RelationV2[period_key#2952L, period_label#2953, period_start#2954, period_end#2955, period_tz#2956] acme_prod.app_dev.vw_period

== Optimized Logical Plan ==
Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, actor_key#2941L, user_id#2776, ... 161 more fields]
+- Join LeftOuter, ((cast(occurred_at#2883 as date) >= period_start#2954) AND (cast(occurred_at#2883 as date) <= period_end#2955))
:- Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, session_id#2762, domain_sessionidx#2763, ... 145 more fields]
: +- Filter ((isnotnull(activity_type#2885) AND isnotnull(occurred_at#2883)) AND ((((NOT (activity_type#2885 = heartbeat) AND cast(activity_type#2885 as string) INSET a, b) AND (occurred_at#2883 > 2026-10-13 09:02:00.404)) AND (occurred_at#2883 <= 2026-10-20 06:36:48.614052)) AND NOT coalesce(is_automated#2945, false)))
: +- Relation acme_prod.app_dev.activity_events[load_date#2756,year#2757,month#2758,day#2759,app_id#2760,page_view_id#2761,session_id#2762,domain_sessionidx#2763,... 183 more fields] parquet
+- RelationV2[period_key#2952L, period_start#2954, period_end#2955] acme_prod.app_dev.vw_period

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- == Initial Plan ==
Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, actor_key#2941L, user_id#2776, ... 161 more fields]
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, ((cast(occurred_at#2883 as date) >= period_start#2954) AND (cast(occurred_at#2883 as date) <= period_end#2955)), true
:- Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, session_id#2762, domain_sessionidx#2763, ... 145 more fields]
: +- Filter ((((if (isnotnull(_databricks_internal_edge_computed_column_skip_row#2979)) (_databricks_internal_edge_computed_column_skip_row#2979 = false) else isnotnull(raise_error(DELTA_SKIP_ROW_COLUMN_NOT_FILLED, map(keys: [], values: []), NullType)) AND isnotnull(occurred_at#2883)) AND (occurred_at#2883 > 2026-10-13 09:02:00.404)) AND (occurred_at#2883 <= 2026-10-20 06:36:48.614052)) AND NOT coalesce(is_automated#2945, false))
: +- FileScan parquet acme_prod.app_dev.activity_events[load_date#2756,app_id#2760,page_view_id#2761,session_id#2762,domain_sessionidx#2763,platform_label#2765,etl_tstamp#2766,collected_at#2767,... 146 more fields] Batched: false, DataFilters: [isnotnull(occurred_at#2883), (occurred_at#2883 > 2026-10-13 09:02:00.404), (occurred_at..., Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[s3://example-bucket/dev/tenant/master-data/activity_events], PartitionFilters: [isnotnull(activity_type#2885), NOT (activity_type#2885 = heartbeat), cast(activity_type#2885 as string) I..., PushedFilters: [IsNotNull(occurred_at), GreaterThan(occurred_at,2026-10-13 09:02:00.404), LessThanOrEqual(..., ReadSchema: struct<load_date:date,app_id:string,page_view_id:string,session_id:string,domain_sessionidx...
+- Exchange SinglePartition, EXECUTOR_BROADCAST, [plan_id=726]
+- Project [period_key#2952L, period_start#2954, period_end#2955]
+- RemoteSparkConnectScan `acme_prod`.`app_dev`.`vw_period`[period_key#2952L, period_start#2954, period_end#2955] PushedPredicates: [period_start IS NOT NULL, period_end IS NOT NULL], PushedAggregates: [], PushedGroupBys: [], PushedLimits: [], PushedSortOrders: [], PushedSamples: [], PushedJoinedRelations: [], RuntimeFilters: []

== Optimizer Statistics (table names per statistics state) ==
missing = activity_events
partial = 
full = 
Corrective actions: consider running the following command on all tables with missing or partial statistics
ANALYZE TABLE <table-name> COMPUTE STATISTICS FOR ALL COLUMNS

vw_period is not expanded. It appears as RelationV2 in the analyzed plan, and in the physical plan it is read by RemoteSparkConnectScan on the broadcast side of the BroadcastNestedLoopJoin. This query runs fine on its own.

With localCheckpoint(eager=True) on the same DataFrame:

== Parsed Logical Plan ==
LogicalRDD [load_date#3305, year#3306, month#3307, day#3308, app_id#3309, page_view_id#3310, actor_key#3311L, user_id#3312, ... 161 more fields], false

== Analyzed Logical Plan ==
load_date: date, year: int, month: int, day: int, app_id: string, page_view_id: string, actor_key: bigint, user_id: string, ... 161 more fields
LogicalRDD [load_date#3305, year#3306, month#3307, day#3308, app_id#3309, page_view_id#3310, actor_key#3311L, user_id#3312, ... 161 more fields], false

== Optimized Logical Plan ==
LogicalRDD [load_date#3305, year#3306, month#3307, day#3308, app_id#3309, page_view_id#3310, actor_key#3311L, user_id#3312, ... 161 more fields], false

== Physical Plan ==
Scan ExistingRDD[load_date#3305,year#3306,month#3307,day#3308,app_id#3309,page_view_id#3310,actor_key#3311L,user_id#3312,... 161 more fields]

The plan becomes LogicalRDD / Scan ExistingRDD. There is no view node and no RemoteSparkConnectScan, and the Step 2 query on top of it runs normally.

[Jen_Evangelista]

Hi harsh120, building on data_pulse's remote-scan observation, I would use localCheckpoint's success as a plan boundary to investigate, rather than proof that MAP is unsupported or that a particular DBR fix exists.

Keep the same compute and run-as identity, then add the second query's pieces incrementally: required base columns, period_key, CONCAT/UPPER/date expressions, and finally MAP plus the actor filter. Trigger the same isEmpty action after each addition and retain the failing plan. This should narrow which change exposes the remote scan failure.

One caution with the workaround: Spark documents local checkpoints as executor-cache-backed and unreliable. Test executor loss and rerun behavior before treating this as a recovery boundary. I would avoid broadening base-table permissions just to eliminate the scan.

For support, retain the exact DBR build, access mode, sanitized view definitions and paired failing/checkpointed plans. I cannot confirm a published fix for this exact case.

References:
https://docs.databricks.com/aws/en/compute/single-user-fgac
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.loca... 

[harsh120]

I also asked Databricks Genie about it, and it described this as a likely bug. I know its answers are not reliable, but it suggests the behavior is unexpected.

For now we dowgraded to DBR 16.4