Using generative AI to generate descriptions

[monke64]

Hi all, I would like to check if there is a function in databricks notebook that allows users to get AI generated descriptions of dataframes within the session similar to the AI generated descriptions for tables? This is because I am wondering it it would be possible to iteratively go through multiple dataframes to generate and store their descriptions. Any help would be much appreciated.

[vikasgautam]

You can use ai_query function to achieve this in SQL. It takes an LLM endpoint and a prompt as input.

Here's an example with DBRX and a specific Indian manufacturing industry-related prompt.

create

or replace function ai_create_table_metadata (table_name STRING) returns string return ai_query(

'databricks-dbrx-instruct',

CONCAT(

'Generate a short business description for the below table name in the context of the Indian Manufacturing Industry:: ',

table_name ));

select

table_name,

ai_create_table_metadata(table_name) as table_desc

from

system.information_schema.tables

where

table_name = 'your_table_name';