PII tags in Spark Declarative Pipelines

bi_123
New Contributor III

I need to add PII tags at both the table and column levels for a streaming table created using Spark Declarative Pipelines.

I tried applying Unity Catalog tags with the following code inside the SDP Python pipeline:

spark.sql(f"""
ALTER TABLE {table_name}
SET TAGS ({tags_sql})
""")

However, this fails with the following error:

UNSUPPORTED_SPARK_SQL_COMMAND
'${command}' is not supported in spark.sql("...") API in SDP Python.
Supported command: ${supportedCommands}.

What is the correct way to define or apply PII tags for tables and columns created by Spark Declarative Pipelines?