Using generative AI to generate descriptions

[monke64]

Hi all, I would like to check if there is a function in databricks notebook that allows users to get AI generated descriptions of dataframes within the session similar to the AI generated descriptions for tables? This is because I am wondering it it would be possible to iteratively go through multiple dataframes to generate and store their descriptions. Any help would be much appreciated.

[vikasgautam]

You can use ai_query function to achieve this in SQL. It takes an LLM endpoint and a prompt as input.

Here's an example with DBRX and a specific Indian manufacturing industry-related prompt.

create

or replace function ai_create_table_metadata (table_name STRING) returns string return ai_query(

'databricks-dbrx-instruct',

CONCAT(

'Generate a short business description for the below table name in the context of the Indian Manufacturing Industry:: ',

table_name ));

select

table_name,

ai_create_table_metadata(table_name) as table_desc

from

system.information_schema.tables

where

table_name = 'your_table_name';

[Nidhig631]

Thanks @vikasgautam  for sharing this solution.

[michael365]

You can also use Genie Code (which was named Assistent before) and put in some instructions as below. This would then be used and applied once the developer uses the Genie Code.

Or alternatively you can create an Agent Skill and put those instructions with the Agent skill file. In addition you can define some key words for the Skill like "generate descriptions" which enables Genie Code to perform the instructions once you enter those two words in a prompt.

In general there is high potential with Genie Code and Agent Skills to automatically:

• check Code in terms of coding guidelines
• check for logic and usage of typical coding instructions
• add documentation
• etc.

## Header documentation

Add a "--" and 50 "=" characters as first line to code
Add a "--" and name of user as author to header as comment 
Add a "--" and timestamp including Day, Date, Time of last updated to header as comment
Adda "-" and  50 "=" characters after author and timestamp
## General
Use consistent and descriptive identifiers and names.
Make judicious use of white space and indentation to make code easier to read.
Store ISO 8601 compliant time and date information (YYYY-MM-DDTHH:MM:SS.SSSSS)
Include comments in SQL code where necessary.
Names must begin with a letter and may not end with an underscore.
Only use letters, numbers and underscores in names.