data_pulse
New Contributor III

yesterday

@25Tabs 

Adding further to above replies, for this use case there aren't any native API calls yet available to generate the comments, but another work around could be:

  • Use IDE's like VS Code with GitHub Copilot for Python Extension enabled. Copilot uses column names, types, nearby code, and similar repository models to suggest comments.
  • Enable VS Code inline suggestions (Check if this is been enabled in vscode): "editor.inlineSuggest.enabled": true​
  • Maintain metadata of the table (Name, Columns, data types, comments etc as kind of data model), basically it's like source of truth of all table related metadata version controlled. It could be in python or yaml files.
  • Then the copilot extension auto suggests table comments when new table is created/editing existing table model file like this, can accept/modify the text slightly from the suggested comments (like below).table_comments_auto_generated.png
  • Commit these changes and when CD is done to any workspace, either can have a mechanism within the code itself to update Table Meta data which reads this data model file and update's table comments/ Ad-hoc one time script to run ALTER table table_name ALTER COLUMN column_name COMMENT with comments from above data model file.

This gives version controlled Column comments and ability to tweak them when required and Govern those aswell.

Another approach on the fly update directly as one of activity:

  • Identify the tables missing columns from information_schema.columns where comment is NULL.
  • Use ai_query to generate the comments as 
SELECT
    table_catalog,
    table_schema,
    table_name,
    column_name,
    data_type,
    ai_query(
      'system.ai.meta-llama-3-3-70b-instruct',
      CONCAT(
        'Generate a concise Unity Catalog column description. ',
        'Return only the description, no quotes or additional explanation. ',
        'Do not invent business meaning that cannot reasonably be inferred. ',
        'If the meaning is ambiguous, say what the column contains without ',
        'assuming additional business semantics. ',
        'Table: ', table_name,
        'Column: ', column_name,
        'Data type: ', data_type
      ),
      modelParameters => named_struct(
        'temperature', 0.0,
        'max_tokens', 80
      )
    ) AS proposed_comment
FROM columns_to_document​
  • Then an update script with results from above query to dynamically apply missing comments for all the tables with ALTER COLUMN Comment mechanism.
  • If needed, can go a step further to validate the ai_query generated column comments with ai_decide (currently in beta version) as quality gate to identify only high confidence score related comments to dynamically update.

 

0 Kudos