Hi.

I've been facing this problem for the last couple of days.

We're experiencing intermittent failures with the error [UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name '_metadata' cannot be resolved. SQLSTATE: 42703 when running MERGE operations on Serverless compute. The same code works consistently on job clusters.

We're experiencing intermittent failures with the error [UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name '_metadata' cannot be resolved. SQLSTATE: 42703 when running MERGE operations on Serverless compute. The same code works consistently on Job Clusters.

Already tried this about the delta.enableRowTracking issue: https://community.databricks.com/t5/get-started-discussions/cannot-run-merge-statement-in-the-notebo...

Context:
Our ingestion pipeline reads Parquet files from a landing zone and merges them into Delta raw tables. We use the _metadata.file_path virtual column to track source files in a Sys_SourceFile column.

Code Pattern:

# Read parquet
df_landing = spark.read.format('parquet').load(landing_path)

# Add system columns including Sys_SourceFile from _metadata
df = df.withColumn('Sys_SourceFile', col('_metadata.file_path'))

# Create temp view
df.createOrReplaceTempView('landing_data')

# Execute MERGE
spark.sql("""
    MERGE INTO target_table AS raw
    USING landing_data AS landing
    ON landing.pk = raw.pk
    WHEN MATCHED AND landing.Sys_Hash != raw.Sys_Hash
    THEN UPDATE SET ...
    WHEN NOT MATCHED BY TARGET
    THEN INSERT ...
""")

Testing & Findings:

_metadata is available after reading to df_landing.

_metadata is available inside the function that adds system columns.

Same table, same parameters, different results:

• Table A - Fails on Serverless

• Table B—with the same config, Works on Serverless

• Both tables have identical delta.enableRowTracking = true

• Both use the same code path

Job Cluster: All tables work consistently.

delta.enableRowTracking: found the community post above suggesting this property causes the issue, but we have tables with enableRowTracking = true that work fine on Serverless, while others with the same property fail.

Key Observations:

• The _metadata virtual column is available at the DataFrame level but gets "lost" somewhere in the execution plan when passed through createOrReplaceTempView() to SQL MERGE.

• The error only manifests at MERGE execution time, not when adding the column with withColumn()

• Behavior is non-deterministic... same code, same config, different tables, different results

• Serverless uses Spark Connect, which "defers analysis and name resolution to execution time"—this seems related, but doesn't explain the inconsistency

Is there a way to work around this? And a solid understanding of why this happens?