Best Practices for Filling Table Information for Databricks Assistant

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi everyone,

I’m looking for advice on the best practices for populating table information in Databricks to ensure that non-SQL users (such as PMs and Marketing teams) can easily query the tables using Databricks Assistant.

Specifically, I have a few questions:

1. Key Columns for Joins: Should I include information about key column(s) often used for joins in the table comment or in the column comment? Which approach would be more beneficial for users and for the Databricks Assistant to understand the relationships between tables?

2. Effectiveness of Comments: Once I add or edit comments (descriptions) for tables and columns, how long does it take for Databricks Assistant to take in the edited context? Is there a delay, and if so, how long is it?

3. Understanding Table Relationships: Does Databricks Assistant actually analyze notebooks to determine which tables are frequently joined together? It seems to struggle with this aspect even when I provide guided context. Is there a way to improve its understanding of these relationships?

Any insights or recommendations you can provide would be greatly appreciated!

Thanks in advance!