Problem:
I’m getting the below error when performing a MERGE (or any other DML command) on a Delta table with a row filter in Databricks.
Error: Cannot find column index for attribute 'account_id#48219' in: Map(transaction_id#47260 -> 5, file_path#47253 -> 0, numPhysicalRecords#49686L -> 3, defaultRowCommitVersion#47262L -> 7, version#47265L -> 9, size#47255L -> 1, account_number#48983 -> 10, baseRowId#47261L -> 6, transaction_tags#47259 -> 4, modificationTime#47256L -> 2, partitionValues_parsed#47264 -> 😎
I’ve verified that account_id is present in the table. For some reason, it is not able to access it in both cases where a row filter allows/blocks data from the user.
Details:
I’ve created a minimal repro that can demonstrate the issue, please find it below.
merge operation:
target_txns.alias("target") \
.merge(source_txns.alias("source"), merge_cond) \
.whenMatchedUpdateAll() \
.whenNotMatchedInsertAll() \
.execute()
The target table has this row filter:
CREATE FUNCTION fin_data.customer_db.filter_account(acc_id STRING) RETURNS BOOLEAN
RETURN acc_id IN (
SELECT account_id
FROM fin_data.customer_db.authorized_accounts
WHERE email = current_user()
);
facing this error, Any leads/help on this would be appreciated.
