Hi Hill22,

Thanks, this is very helpful. I think the lightweight approach using Databricks SQL Queries, domain-based folders, metadata headers, and an archive process makes a lot of sense for our current situation.

Our main concern is that the number of queries may grow significantly over time, so simply organizing them into folders might eventually become difficult to navigate.

Would you recommend maintaining a lightweight index/catalog in addition to the folder structure, or do you think a good naming convention, folder structure, and metadata for each query are sufficient for managing a large number of queries?

Also, for the 90-day rule, would you treat it as a review trigger rather than automatically archiving the query?

Hi @HILL22 ,

For a simple and maintainable process, the best approach is to use the native Databricks SQL (DBSQL) Queries feature combined with Tags and Workspace Folders.

Here is how you can set up that workflow:

1. Save them directly in the Databricks SQL > Queries section. Create a dedicated folder structure in your Workspace (e.g., Shared/Ad-hoc_Requests/Finance or Shared/Ad-hoc_Requests/Marketing) to keep them out of personal user folders.

2. Use the native Tags feature on the query in the DBSQL UI. You can create a standardized tagging system like:

• Department: finance, sales

• Requester: req:john_doe

• Status: active, one-off, deprecated

Pro-tip: Put a standard comment block at the very top of the SQL code for the details:

SQL

 

/*
Requester: John Doe (Marketing)
Ticket: JIRA-1234
Purpose: Extract Q3 churned users for email campaign
Owner: Data Team
*/
SELECT ...

3. Because you are using tags, you can easily filter the Queries UI by the one-off or deprecated tags and delete them quarterly. You can also look at the "Last Run" column in the Queries list to quickly spot queries that haven't been touched in months.

4. If your team wants version control later, you can save these as .sql files inside Databricks Git Folders (Repos). However, if your main goal is simplicity right now, just stick to the DBSQL Queries UI with strict tagging and folder management.

Thanks for the great suggestions! I hadn't considered using Query Tags in this way. Combining tags with folders and naming conventions seems like a practical approach for keeping the queries organized as they grow. Really appreciate the helpful advice!

The catalog table generally adds maintenance overhead in manual cases as it can drift out of sync with actual queries if managed manually. Databricks capabilities might be enough as you can use Workspace search (searches across query names and content including SQL comments), Folder browsing works fine up to 100-200 queries per folder, System tables automatically tracks who ran what and when and SQL comments contain all the metadata you need

You can start simple and add complexity only if needed. 

1. Start with: Folders + naming convention + SQL comment metadata
2. Use workspace search when you need to find something
3. Query system tables periodically to identify unused queries & usage statistics
4. Add a catalog table or an app that takes the request info & load the catalog table or you can run it via workflow when Workspace search isn't sufficient & you get complex reporting requests

You can skip additional table unless you're managing thousands of queries or need sophisticated reporting/analytics on the requests as the lightweight approach (folders + naming + comments) is usually sufficient for direct cases.

90-day rule is a trigger to highlight unused queries. You can archive it using workflows after a review.

Thanks for the detailed suggestions! This is very helpful. I think the lightweight approach with domain-based folders, naming conventions, metadata, and an archive process fits our use case very well. I really like the idea of keeping the management simple within Databricks without introducing unnecessary overhead.