Comments not updating on a SDP streaming table

liquibricks · 3 weeks ago

We have a pipeline in a job which dynamically creates a set of streaming tables based on a list of kafka topics like this:

# inside a loop

@DP.table(name=table_name, comment=markdown_info)

def topic_flow(topic_name=topic_name):

return (

read_from_kafka(important_parameters)

)

But it seems as if there is a change to some metadata inside `markdown_info` that the underlying table is not updated when the pipeline restarts. Is this known / expected behaviour? Is there any way to update the comment without having to recreate the table?

K_Anudeep · 3 weeks ago

Hello @liquibricks

Yes, it is a known and expected behaviour that changing the comment value passed to the @DP.table decorator (or similar DLT table decorators) does not automatically update the physical table's comment in the underlying catalogue when the pipeline is restarted or updated, if the table already exists. Once a streaming table is created, metadata set at creation such as the table comment, remains attached to the table unless explicitly changed using supported metadata-altering operations

In SDP/DLT streaming tables, Databricks blocks direct DDL changes to schema/properties (including table comments) outside the pipeline to maintain metadata consistency with the pipeline definition. If you try COMMENT ON TABLE, Databricks throws an error and tells you to use CREATE OR REFRESH instead.

Doc: https://kb.databricks.com/en_US/delta-live-tables/unable-to-modify-comments-on-streaming-tables

So the only way to fix this is to update the comment in the pipeline code instead then run a pipeline update for it two work

Anudeep

liquibricks · 3 weeks ago

Thanks for the reply. In our case the comment (in "markdown_info") is updated as part of the pipeline, however the updated value is not reflected in Unity Catalog. What did you mean by "update the comment in the pipeline code instead then run a pipeline update"?

liquibricks · 3 weeks ago

Ah, my code is correct. There was just a mistake further up when producing the comments that lead me down the wrong path. Comments (and metadata) are correctly updated as expected!

View solution in original post