We have a pipeline in a job which dynamically creates a set of streaming tables based on a list of kafka topics like this: # inside a loop @DP.table(name=table_name, comment=markdown_info) def topic_flow(topic_name=topic_name): ...
We are testing an ingestion from kafka to databricks using a streaming table. The streaming table was created by a DAB deployed to "production" which runs as a service principal. This means the service principal is the "owner" of the table.We now wan...
Ah, my code is correct. There was just a mistake further up when producing the comments that lead me down the wrong path. Comments (and metadata) are correctly updated as expected!
Thanks for the reply. In our case the comment (in "markdown_info") is updated as part of the pipeline, however the updated value is not reflected in Unity Catalog. What did you mean by "update the comment in the pipeline code instead then run a pipel...
Thanks for your reply, Nayan.I'll try option 1 and report back later.Is option 2 possible? As per the second error message, it does not appear as if we can change the owner on streaming tables.Option 3 would be too complex and time-consuming, but tha...
