topic Re: Table does not support batch scan. in Data Engineering

Table does not support batch scan.

harsh120 — Mon, 19 Oct 2026 13:15:27 GMT

After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error:

[UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported:
Table `app_dev`.`vw_period` does not support batch scan. Please check the
current catalog and namespace to make sure the qualified table name is
expected, and also check the catalog implementation which is configured
by "spark.sql.catalog". SQLSTATE: 0A000

vw_period is a regular Unity Catalog view, and the table under it is a plain external Delta table, not a streaming table, materialized view or pipeline output. The same views, data and code ran without problems on 16.4.

Table and view Defination

-- external Delta table, about 20 rows in total
CREATE TABLE acme_prod.app_dev.calendar_entries (
  entry_id BIGINT, period_key BIGINT, period_label STRING,
  period_start DATE, period_end DATE, period_tz STRING
  -- other columns omitted
) USING DELTA;

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_period AS
SELECT DISTINCT period_key, period_label, period_start, period_end, period_tz
FROM acme_prod.app_dev.calendar_entries;

-- external Delta table, partitioned by activity_type; the real one has about 170 columns
CREATE TABLE acme_prod.app_dev.activity_events (
  activity_id STRING, actor_key BIGINT, activity_type STRING,
  platform_label STRING, session_id STRING,
  occurred_at TIMESTAMP, is_automated BOOLEAN
  -- other columns omitted
) USING DELTA PARTITIONED BY (activity_type);

CREATE OR REPLACE VIEW acme_prod.app_dev.vw_activity_events AS
SELECT e.*, COALESCE(p.period_key, CAST(-1 AS BIGINT)) AS period_key
FROM acme_prod.app_dev.activity_events e
LEFT JOIN acme_prod.app_dev.vw_period p
  ON CAST(e.occurred_at AS DATE) BETWEEN p.period_start AND p.period_end
WHERE e.activity_type <> 'heartbeat';

What works and what fails

The first read works on DBR 18, and isEmpty() and rdd.isEmpty() both return normally on it:

src=spark.sql("""
  SELECT * FROM acme_prod.app_dev.vw_activity_events
  WHERE activity_type IN ('a', 'b')
    AND occurred_at >  TIMESTAMP '2026-10-13 09:02:00'
    AND occurred_at <= TIMESTAMP '2026-10-19 12:46:00'
    AND COALESCE(is_automated, false) = false
""")
src.createOrReplaceTempView("events_incr")

The next step is where it fails:

out = spark.sql("""
  SELECT CONCAT('X_', activity_id) AS event_key, actor_key,
         UPPER(activity_type) AS activity_type, CAST(occurred_at AS DATE) AS event_date,
         period_key,
         MAP('platform_label', platform_label, 'session_id', session_id) AS props
  FROM events_incr
""").filter("actor_key IS NOT NULL")

out.isEmpty()   # raises the error above

The plan for the first query is below. Under the outer view, vw_period is not expanded into its SELECT DISTINCT. It shows up as RelationV2 and is read by RemoteSparkConnectScan, while the outer view is expanded normally.

== Parsed Logical Plan ==
'Project [*]
+- 'Filter ((activity_type IN (...) AND (occurred_at > 2026-10-13 09:02:00.404))
            AND ((occurred_at <= 2026-10-19 12:46:43.923587)
                 AND (COALESCE(is_automated, false) = false)))
   +- 'UnresolvedRelation [app_dev, vw_activity_events], [], false

== Analyzed Logical Plan ==
Project [...]
+- Filter ((activity_type IN (...) AND (occurred_at > ...)) AND ((occurred_at <= ...) AND (coalesce(is_automated, false) = false)))
   +- SubqueryAlias acme_prod.app_dev.vw_activity_events
      +- View (acme_prod.app_dev.vw_activity_events, [...])
         +- Project [...]
            +- Filter NOT (activity_type = heartbeat)
               +- Join LeftOuter, between(cast(occurred_at as date), period_start, period_end)
                  :- SubqueryAlias acme_prod.app_dev.activity_events
                  :  +- Relation acme_prod.app_dev.activity_events[...] parquet
                  +- SubqueryAlias p
                     +- SubqueryAlias acme_prod.app_dev.vw_period
                        +- RelationV2[period_key, period_label, period_start, period_end, period_tz] acme_prod.app_dev.vw_period

== Optimized Logical Plan ==
Project [...]
+- Join LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end))
   :- Project [...]
   :  +- Filter (activity_type INSET (...) AND occurred_at > ... AND occurred_at <= ... AND NOT coalesce(is_automated, false))
   :     +- Relation acme_prod.app_dev.activity_events[...] parquet
   +- RelationV2[period_key, period_start, period_end] acme_prod.app_dev.vw_period

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, ((cast(occurred_at as date) >= period_start) AND (cast(occurred_at as date) <= period_end)), true
   :- FileScan parquet acme_prod.app_dev.activity_events[...]
   :     Batched: false
   :     Location: PreparedDeltaFileIndex(1 paths)[s3://example-bucket/dev/tenant/master-data/activity_events]
   :     PartitionFilters: [isnotnull(activity_type), NOT (activity_type = heartbeat), activity_type INSET (...)]
   :     PushedFilters: [IsNotNull(occurred_at), GreaterThan(occurred_at, ...), LessThanOrEqual(occurred_at, ...)]
   +- Exchange SinglePartition, EXECUTOR_BROADCAST
      +- Project [period_key, period_start, period_end]
         +- RemoteSparkConnectScan acme_prod.app_dev.vw_period[period_key, period_start, period_end]
              PushedPredicates: [period_start IS NOT NULL, period_end IS NOT NULL]
              Runtime row count reported for this side: 5
              Size: 160 bytes

What I tried

Setting spark.sql.ansi.enabled to false made no difference.
Recreating both views, and using fully qualified names, made no difference. DESCRIBE EXTENDED on both views shows the correct catalog and namespace.
Calling localCheckpoint(eager=True) on the first DataFrame, before the second step, lets the whole job run on DBR 18. That is our workaround for now.
A second job joins another Delta table to vw_activity_events and adds a transform() over an array column in the same query. It fails the same way. Splitting the join from the transform() with a checkpoint avoids it. I haven’t isolated which expression triggers it.

Is this a known issue in DBR 18? Why is vw_period planned as RelationV2 and read through RemoteSparkConnectScan instead of being expanded, and what makes the second query’s scan fail when the first query’s succeeds? Is there a setting or recommended pattern to use until a fix is out?

Re: Table does not support batch scan.

data_pulse — Mon, 19 Oct 2026 15:39:44 GMT

@harsh120

I tried reproducing this with a similar setup using nested permanent Unity Catalog views, including SELECT DISTINCT, a date-range LEFT JOIN, a temporary view and the subsequent MAP()/filter transformations with isEmpty().

Both steps passed on Databricks Serverless (Spark 4.2.0), including with ANSI enabled. The execution plans expanded the UC views into the underlying Delta tables, using PhotonScan and PhotonBroadcastNestedLoopJoin.

One notable difference from your DBR 18 plan is RemoteSparkConnectScan / RelationV2 for vw_period, rather than expansion into the underlying Delta table.

Databricks documents RemoteSparkConnectScan in the context of fine-grained access control on Dedicated compute.

Since localCheckpoint(eager=True) works, I suspect the issue may involve how DBR 18 replans the nested view through this remote scan during Step 2, rather than the MAP() expression or ANSI mode itself.

Few questions to understand further:

1. Is this running on Dedicated compute, and does the job identity have SELECT on the underlying tables or only on the UC views?

2. If possible, could you share the Step 2 explain() plan both with and without the checkpoint? That would confirm whether RemoteSparkConnectScan disappears after materialisation.

Re: Table does not support batch scan.

Jen_Evangelista — Mon, 19 Oct 2026 22:34:55 GMT

Hi harsh120, building on data_pulse's remote-scan observation, I would use localCheckpoint's success as a plan boundary to investigate, rather than proof that MAP is unsupported or that a particular DBR fix exists.

Keep the same compute and run-as identity, then add the second query's pieces incrementally: required base columns, period_key, CONCAT/UPPER/date expressions, and finally MAP plus the actor filter. Trigger the same isEmpty action after each addition and retain the failing plan. This should narrow which change exposes the remote scan failure.

One caution with the workaround: Spark documents local checkpoints as executor-cache-backed and unreliable. Test executor loss and rerun behavior before treating this as a recovery boundary. I would avoid broadening base-table permissions just to eliminate the scan.

For support, retain the exact DBR build, access mode, sanitized view definitions and paired failing/checkpointed plans. I cannot confirm a published fix for this exact case.

References:
https://docs.databricks.com/aws/en/compute/single-user-fgac
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.localCheckpoint.html

Re: Table does not support batch scan.

harsh120 — Tue, 20 Oct 2026 07:16:33 GMT

1. Is this running on Dedicated compute, and does the job identity have SELECT on the underlying tables or only on the UC views?

Yes, it is Dedicated access mode on DBR 18 (Spark 4.1.0, also tested on DBR 19, spark 4.2). The identity running the query owns the views and the underlying Delta tables, and also has catalog-level privileges (SELECT and ALL PRIVILEGES) through a group. So it has direct SELECT on the base tables, not only on the views.

2. If possible, could you share the Step 2 explain() plan both with and without the checkpoint? That would confirm whether RemoteSparkConnectScan disappears after materialisation.

Both plans are for the Step 1 read (SELECT * from vw_activity_events), shortened to the first 8 columns of each list. Table and column names are changed.

Without the checkpoint:

== Parsed Logical Plan ==
'Project [*]
+- 'Filter (('activity_type IN (a, b) AND ('occurred_at > 2026-10-13 09:02:00.404)) AND (('occurred_at <= 2026-10-20 06:36:48.614052) AND ('COALESCE('is_automated, false) = false)))
+- 'UnresolvedRelation [app_dev, vw_activity_events], [], false

== Analyzed Logical Plan ==
load_date: date, year: int, month: int, day: int, app_id: string, page_view_id: string, actor_key: bigint, user_id: string, ... 161 more fields
Project [load_date#2396, year#2397, month#2398, day#2399, app_id#2400, page_view_id#2401, actor_key#2402L, user_id#2403, ... 161 more fields]
+- Filter ((activity_type#2526 IN (a, b) AND (occurred_at#2524 > 2026-10-13 09:02:00.404)) AND ((occurred_at#2524 <= 2026-10-20 06:36:48.614052) AND (coalesce(is_automated#2563, false) = false)))
+- SubqueryAlias acme_prod.app_dev.vw_activity_events
+- View (`acme_prod`.`app_dev`.`vw_activity_events`, [load_date#2396, year#2397, month#2398, day#2399, app_id#2400, page_view_id#2401, actor_key#2402L, user_id#2403, ... 161 more fields])
+- Project [cast(load_date#2756 as date) AS load_date#2396, cast(year#2757 as int) AS year#2397, cast(month#2758 as int) AS month#2398, cast(day#2759 as int) AS day#2399, cast(app_id#2760 as string) AS app_id#2400, cast(page_view_id#2761 as string) AS page_view_id#2401, cast(actor_key#2941L as bigint) AS actor_key#2402L, cast(user_id#2776 as string) AS user_id#2403, ... 161 more fields]
+- Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, actor_key#2941L, user_id#2776, ... 161 more fields]
+- Filter NOT (activity_type#2885 = heartbeat)
+- Join LeftOuter, between(cast(occurred_at#2883 as date), period_start#2954, period_end#2955)
:- SubqueryAlias e
: +- SubqueryAlias acme_prod.app_dev.activity_events
: +- Relation acme_prod.app_dev.activity_events[load_date#2756,year#2757,month#2758,day#2759,app_id#2760,page_view_id#2761,session_id#2762,domain_sessionidx#2763,... 183 more fields] parquet
+- SubqueryAlias p
+- SubqueryAlias acme_prod.app_dev.vw_period
+- RelationV2[period_key#2952L, period_label#2953, period_start#2954, period_end#2955, period_tz#2956] acme_prod.app_dev.vw_period

== Optimized Logical Plan ==
Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, actor_key#2941L, user_id#2776, ... 161 more fields]
+- Join LeftOuter, ((cast(occurred_at#2883 as date) >= period_start#2954) AND (cast(occurred_at#2883 as date) <= period_end#2955))
:- Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, session_id#2762, domain_sessionidx#2763, ... 145 more fields]
: +- Filter ((isnotnull(activity_type#2885) AND isnotnull(occurred_at#2883)) AND ((((NOT (activity_type#2885 = heartbeat) AND cast(activity_type#2885 as string) INSET a, b) AND (occurred_at#2883 > 2026-10-13 09:02:00.404)) AND (occurred_at#2883 <= 2026-10-20 06:36:48.614052)) AND NOT coalesce(is_automated#2945, false)))
: +- Relation acme_prod.app_dev.activity_events[load_date#2756,year#2757,month#2758,day#2759,app_id#2760,page_view_id#2761,session_id#2762,domain_sessionidx#2763,... 183 more fields] parquet
+- RelationV2[period_key#2952L, period_start#2954, period_end#2955] acme_prod.app_dev.vw_period

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- == Initial Plan ==
Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, actor_key#2941L, user_id#2776, ... 161 more fields]
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, ((cast(occurred_at#2883 as date) >= period_start#2954) AND (cast(occurred_at#2883 as date) <= period_end#2955)), true
:- Project [load_date#2756, year#2757, month#2758, day#2759, app_id#2760, page_view_id#2761, session_id#2762, domain_sessionidx#2763, ... 145 more fields]
: +- Filter ((((if (isnotnull(_databricks_internal_edge_computed_column_skip_row#2979)) (_databricks_internal_edge_computed_column_skip_row#2979 = false) else isnotnull(raise_error(DELTA_SKIP_ROW_COLUMN_NOT_FILLED, map(keys: [], values: []), NullType)) AND isnotnull(occurred_at#2883)) AND (occurred_at#2883 > 2026-10-13 09:02:00.404)) AND (occurred_at#2883 <= 2026-10-20 06:36:48.614052)) AND NOT coalesce(is_automated#2945, false))
: +- FileScan parquet acme_prod.app_dev.activity_events[load_date#2756,app_id#2760,page_view_id#2761,session_id#2762,domain_sessionidx#2763,platform_label#2765,etl_tstamp#2766,collected_at#2767,... 146 more fields] Batched: false, DataFilters: [isnotnull(occurred_at#2883), (occurred_at#2883 > 2026-10-13 09:02:00.404), (occurred_at..., Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[s3://example-bucket/dev/tenant/master-data/activity_events], PartitionFilters: [isnotnull(activity_type#2885), NOT (activity_type#2885 = heartbeat), cast(activity_type#2885 as string) I..., PushedFilters: [IsNotNull(occurred_at), GreaterThan(occurred_at,2026-10-13 09:02:00.404), LessThanOrEqual(..., ReadSchema: struct<load_date:date,app_id:string,page_view_id:string,session_id:string,domain_sessionidx...
+- Exchange SinglePartition, EXECUTOR_BROADCAST, [plan_id=726]
+- Project [period_key#2952L, period_start#2954, period_end#2955]
+- RemoteSparkConnectScan `acme_prod`.`app_dev`.`vw_period`[period_key#2952L, period_start#2954, period_end#2955] PushedPredicates: [period_start IS NOT NULL, period_end IS NOT NULL], PushedAggregates: [], PushedGroupBys: [], PushedLimits: [], PushedSortOrders: [], PushedSamples: [], PushedJoinedRelations: [], RuntimeFilters: []

== Optimizer Statistics (table names per statistics state) ==
missing = activity_events
partial = 
full = 
Corrective actions: consider running the following command on all tables with missing or partial statistics
ANALYZE TABLE <table-name> COMPUTE STATISTICS FOR ALL COLUMNS

vw_period is not expanded. It appears as RelationV2 in the analyzed plan, and in the physical plan it is read by RemoteSparkConnectScan on the broadcast side of the BroadcastNestedLoopJoin. This query runs fine on its own.

With localCheckpoint(eager=True) on the same DataFrame:

== Parsed Logical Plan ==
LogicalRDD [load_date#3305, year#3306, month#3307, day#3308, app_id#3309, page_view_id#3310, actor_key#3311L, user_id#3312, ... 161 more fields], false

== Analyzed Logical Plan ==
load_date: date, year: int, month: int, day: int, app_id: string, page_view_id: string, actor_key: bigint, user_id: string, ... 161 more fields
LogicalRDD [load_date#3305, year#3306, month#3307, day#3308, app_id#3309, page_view_id#3310, actor_key#3311L, user_id#3312, ... 161 more fields], false

== Optimized Logical Plan ==
LogicalRDD [load_date#3305, year#3306, month#3307, day#3308, app_id#3309, page_view_id#3310, actor_key#3311L, user_id#3312, ... 161 more fields], false

== Physical Plan ==
Scan ExistingRDD[load_date#3305,year#3306,month#3307,day#3308,app_id#3309,page_view_id#3310,actor_key#3311L,user_id#3312,... 161 more fields]

The plan becomes LogicalRDD / Scan ExistingRDD. There is no view node and no RemoteSparkConnectScan, and the Step 2 query on top of it runs normally.

Re: Table does not support batch scan.

harsh120 — Tue, 20 Oct 2026 07:23:51 GMT

I also asked Databricks Genie about it, and it described this as a likely bug. I know its answers are not reliable, but it suggests the behavior is unexpected.

For now we dowgraded to DBR 16.4

Re: Table does not support batch scan.

Aravind_Reddy — Thu, 22 Oct 2026 08:05:04 GMT

Hi @harsh120

The RemoteSparkConnectScan in the execution plan is an important clue. On Dedicated compute, Databricks can delegate queries involving fine-grained access controls to serverless compute, which explains why a remote scan may appear. However, since the underlying view and tables have no reported FGAC restrictions, I wouldn't assume this is the root cause without further validation.

Since localCheckpoint(eager=True) resolves the issue by materializing the DataFrame and removing the remote scan from the subsequent plan, I'd compare the failing and successful execution plans and check whether the behavior is specific to DBR 18/19.

As a workaround, consider materializing the intermediate result to a durable staging Delta table if appropriate for the incremental pipeline. Unlike localCheckpoint, this provides a more reliable recovery boundary.

It would be useful to confirm whether Databricks has identified this as a runtime regression and whether a fix or recommended workaround is available.

Re: Table does not support batch scan.

Louis_Frolio — Thu, 22 Oct 2026 19:09:11 GMT

Hello @harsh120, I took a look at both internal and external documentation and here is what I found.

Building on @data_pulse and @Jen_Evangelista: RemoteSparkConnectScan is the key clue, and this doesn't look like MAP(), isEmpty(), or ANSI mode being unsupported on their own.

On Dedicated compute, Databricks delegates part of a query to serverless compute when fine-grained access control is involved, and that delegation shows up in the plan as RemoteSparkConnectScan. The documented triggers are dynamic views, tables with row filters or column masks, materialized views, streaming tables, and views over tables the querying user lacks SELECT on. None of those should apply to vw_period, since your identity owns calendar_entries and has SELECT on it directly. 16.4 inlined the view; 18 and 19 route it remotely while expanding vw_activity_events normally. I couldn't find a release note or doc that explains why a nested view would be treated differently from the outer one.

The error adds a second clue. "Does not support batch scan" is Spark's UNSUPPORTED_FEATURE.TABLE_OPERATION message for a DataSource V2 table object that doesn't implement the readable interface. Your first query got a readable remote table; the second, resolved through the events_incr temp view, got one it couldn't read. Same view, different outcome depending on how the plan is reached. That's a planner inconsistency on the remote-filtering path, which lines up with Jen's point about treating the checkpoint as a plan boundary rather than evidence about any specific expression. I can't confirm a published fix for this case, so I'd open a support ticket with the two explain plans you already have, the exact runtime build string, and the fact that it reproduces on 18 and 19 but not 16.4. One housekeeping item first: DBR 18 now ships updates as dated entries that clusters pick up on restart, so make sure the cluster has been restarted recently and record the build you're on.

Checks that either confirm the mismatch or expose a legitimate trigger:

EXPLAIN SELECT * FROM acme_prod.app_dev.vw_period on its own on the DBR 18 cluster. If a standalone read shows RemoteSparkConnectScan, the routing decision is about the view itself, not the nesting.
SHOW CREATE TABLE on the view, plus acme_prod.information_schema.row_filters and column_masks for calendar_entries, and a look for any ABAC policy at the catalog or schema level that targets a tag on one of its columns. ABAC attaches above the table, so a table can pick up a mask without anyone touching its definition.
system.billing.usage filtered to billing_origin_product = 'FINE_GRAINED_ACCESS_CONTROL' for the job's run-as identity. If the cluster is paying serverless DBUs to read a 20-row view you own, that's a cost problem on top of the failure and good evidence for the ticket.

On workarounds, you have cleaner options than localCheckpoint. Jen's caution stands: local checkpoints live in executor storage and don't survive executor loss. In rough order of how clean they are:

Remove the nesting. vw_period is a SELECT DISTINCT projection, so fold it into vw_activity_events as a subquery over calendar_entries. Base tables you have SELECT on don't take the remote path.
Skip the temp view hop and keep chaining on src (src.selectExpr(...).filter(...)). The failure appears when the plan goes through the temp view a second time, so this is a cheap test, though I haven't verified it.
If you need a materialization boundary, write the slice to a Delta staging table, or use checkpoint(eager=True) with sc.setCheckpointDir pointed at a Unity Catalog volume, which is supported on Dedicated compute and restart-safe.
Run the minimal repro on Standard access mode or serverless. Standard enforces fine-grained access natively with no remote scan, and data_pulse's repro passed on serverless. If it passes there and fails on Dedicated, that isolates the issue to the remote-filtering path.

Downgrading to 16.4 is a reasonable short-term move; it's LTS through May 2028. I'd still file the ticket, since engineering is the only group that can explain the routing change.

References:

Fine-grained access control on dedicated compute: https://docs.databricks.com/aws/en/compute/single-user-fgac
Dedicated compute requirements and limitations: https://docs.databricks.com/aws/en/compute/dedicated-limitations
Access modes: https://docs.databricks.com/aws/en/compute/configure#access-mode
ABAC requirements and limitations: https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/requirements
Databricks Runtime 18 LTS release notes: https://docs.databricks.com/aws/en/release-notes/runtime/18
Databricks Runtime support lifecycles: https://docs.databricks.com/aws/en/release-notes/runtime/databricks-runtime-ver
DataFrame checkpoints in volumes: https://docs.databricks.com/aws/en/volumes/volume-files#dataframe-checkpoints
PySpark DataFrame.checkpoint: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.checkpoint.html
PySpark DataFrame.localCheckpoint: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.localCheckpoint.html

Regards,
Louis