- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 weeks ago
Hello @harsh120, I took a look at both internal and external documentation and here is what I found.
Building on @data_pulse and @Jen_Evangelista: RemoteSparkConnectScan is the key clue, and this doesn't look like MAP(), isEmpty(), or ANSI mode being unsupported on their own.
On Dedicated compute, Databricks delegates part of a query to serverless compute when fine-grained access control is involved, and that delegation shows up in the plan as RemoteSparkConnectScan. The documented triggers are dynamic views, tables with row filters or column masks, materialized views, streaming tables, and views over tables the querying user lacks SELECT on. None of those should apply to vw_period, since your identity owns calendar_entries and has SELECT on it directly. 16.4 inlined the view; 18 and 19 route it remotely while expanding vw_activity_events normally. I couldn't find a release note or doc that explains why a nested view would be treated differently from the outer one.
The error adds a second clue. "Does not support batch scan" is Spark's UNSUPPORTED_FEATURE.TABLE_OPERATION message for a DataSource V2 table object that doesn't implement the readable interface. Your first query got a readable remote table; the second, resolved through the events_incr temp view, got one it couldn't read. Same view, different outcome depending on how the plan is reached. That's a planner inconsistency on the remote-filtering path, which lines up with Jen's point about treating the checkpoint as a plan boundary rather than evidence about any specific expression. I can't confirm a published fix for this case, so I'd open a support ticket with the two explain plans you already have, the exact runtime build string, and the fact that it reproduces on 18 and 19 but not 16.4. One housekeeping item first: DBR 18 now ships updates as dated entries that clusters pick up on restart, so make sure the cluster has been restarted recently and record the build you're on.
Checks that either confirm the mismatch or expose a legitimate trigger:
EXPLAIN SELECT * FROM acme_prod.app_dev.vw_periodon its own on the DBR 18 cluster. If a standalone read showsRemoteSparkConnectScan, the routing decision is about the view itself, not the nesting.SHOW CREATE TABLEon the view, plusacme_prod.information_schema.row_filtersandcolumn_masksforcalendar_entries, and a look for any ABAC policy at the catalog or schema level that targets a tag on one of its columns. ABAC attaches above the table, so a table can pick up a mask without anyone touching its definition.system.billing.usagefiltered tobilling_origin_product = 'FINE_GRAINED_ACCESS_CONTROL'for the job's run-as identity. If the cluster is paying serverless DBUs to read a 20-row view you own, that's a cost problem on top of the failure and good evidence for the ticket.
On workarounds, you have cleaner options than localCheckpoint. Jen's caution stands: local checkpoints live in executor storage and don't survive executor loss. In rough order of how clean they are:
- Remove the nesting.
vw_periodis aSELECT DISTINCTprojection, so fold it intovw_activity_eventsas a subquery overcalendar_entries. Base tables you have SELECT on don't take the remote path. - Skip the temp view hop and keep chaining on
src(src.selectExpr(...).filter(...)). The failure appears when the plan goes through the temp view a second time, so this is a cheap test, though I haven't verified it. - If you need a materialization boundary, write the slice to a Delta staging table, or use
checkpoint(eager=True)withsc.setCheckpointDirpointed at a Unity Catalog volume, which is supported on Dedicated compute and restart-safe. - Run the minimal repro on Standard access mode or serverless. Standard enforces fine-grained access natively with no remote scan, and data_pulse's repro passed on serverless. If it passes there and fails on Dedicated, that isolates the issue to the remote-filtering path.
Downgrading to 16.4 is a reasonable short-term move; it's LTS through May 2028. I'd still file the ticket, since engineering is the only group that can explain the routing change.
References:
- Fine-grained access control on dedicated compute: https://docs.databricks.com/aws/en/compute/single-user-fgac
- Dedicated compute requirements and limitations: https://docs.databricks.com/aws/en/compute/dedicated-limitations
- Access modes: https://docs.databricks.com/aws/en/compute/configure#access-mode
- ABAC requirements and limitations: https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/requirements
- Databricks Runtime 18 LTS release notes: https://docs.databricks.com/aws/en/release-notes/runtime/18
- Databricks Runtime support lifecycles: https://docs.databricks.com/aws/en/release-notes/runtime/databricks-runtime-ver
- DataFrame checkpoints in volumes: https://docs.databricks.com/aws/en/volumes/volume-files#dataframe-checkpoints
- PySpark DataFrame.checkpoint: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.chec...
- PySpark DataFrame.localCheckpoint: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.loca...
Regards,
Louis