topic Re: How to manage SQL queries for business data extraction in Databricks? in Warehousing & Analytics

How to manage SQL queries for business data extraction in Databricks?

HILL22 — Tue, 11 Aug 2026 09:13:38 GMT

Hi everyone,

Our data team frequently receives data extraction requests from business teams. Most requests are relatively simple and are handled by writing SQL queries in Databricks.

We are looking for a good way to manage these SQL queries as the number of requests grows.

For example:

Where should these SQL queries be stored?
Should they be managed in Git, Databricks SQL, or another tool?
How do you organize and name queries?
How do you track the requester, purpose, owner, and status of each query?
How do you handle old or unused queries?

We don't necessarily want to turn every request into a view, table, or dashboard. The main goal is to establish a simple and maintainable process for managing business data extraction SQL.

What is the recommended approach or best practice for this in Databricks?

Thanks!

Re: How to manage SQL queries for business data extraction in Databricks?

balajij8 — Tue, 11 Aug 2026 09:35:23 GMT

Hi Hill22,

You can manage ad-hoc business data extraction requests inside Databricks without taking on heavy DevOps overhead at this stage. Keep the SQL queries inside Databricks SQL until an extraction actually requires to be elevated into a managed production asset.

You can follow below

Storage and Workspace Structure

You can save the SQL queries directly as Databricks SQL Queries rather than notebooks or Git repositories. For pure ad-hoc extractions, Git integration may create unnecessary friction for analysts who need to pull a quick answer for business stakeholders. Organize them under /Workspace/Shared/Data Requests/ by department (/Shared/Data Requests/Finance/). Keep a dedicated /Archive/ subfolder within each domain directory so completed or old queries can be moved out of the primary workspace view.

Naming Pattern

You can adopt a strict, sortable naming format so anyone browsing the folder instantly understands the context without having to open the query file.

[Task]_[ShortDescription]

Metadata Header

You can use a standard SQL comment block at the very top of every saved query rather than maintaining a separate tracking sheet or registry table that will inevitably drift out of date

Requester: Name / Email
Purpose: Problem this query answers
Ticket Ref: JIRA / ServiceNow ticket link or ID
Owner: Analyst who built the query
Status: Active, Recurring or Archived

Auditing and Lifecycle Tracking via System Tables

Instead of manually tracking usage leverage Unity Catalog audit execution history across the workspace to drive lifecycle decisions.

Dormant queries - Saved queries that haven't been run in 90+ days, making them candidates to be moved into the /Archive/ folder.

Keep queries in domain-structured folders with structured comment headers as it handles the vast majority of extraction requests without building complexity.

Re: How to manage SQL queries for business data extraction in Databricks?

HILL22 — Tue, 11 Aug 2026 10:05:29 GMT

Thanks, this is very helpful. I think the lightweight approach using Databricks SQL Queries, domain-based folders, metadata headers, and an archive process makes a lot of sense for our current situation.

Our main concern is that the number of queries may grow significantly over time, so simply organizing them into folders might eventually become difficult to navigate.

Would you recommend maintaining a lightweight index/catalog in addition to the folder structure, or do you think a good naming convention, folder structure, and metadata for each query are sufficient for managing a large number of queries?

Also, for the 90-day rule, would you treat it as a review trigger rather than automatically archiving the query?

Re: How to manage SQL queries for business data extraction in Databricks?

ShamenParis — Tue, 11 Aug 2026 10:10:57 GMT

Hi @HILL22 ,

For a simple and maintainable process, the best approach is to use the native Databricks SQL (DBSQL) Queries feature combined with Tags and Workspace Folders.

Here is how you can set up that workflow:

1. Save them directly in the Databricks SQL > Queries section. Create a dedicated folder structure in your Workspace (e.g., Shared/Ad-hoc_Requests/Finance or Shared/Ad-hoc_Requests/Marketing) to keep them out of personal user folders.

2. Use the native Tags feature on the query in the DBSQL UI. You can create a standardized tagging system like:

Department: finance, sales
Requester: req:john_doe
Status: active, one-off, deprecated

Pro-tip: Put a standard comment block at the very top of the SQL code for the details:

SQL

/*
Requester: John Doe (Marketing)
Ticket: JIRA-1234
Purpose: Extract Q3 churned users for email campaign
Owner: Data Team
*/
SELECT ...

3. Because you are using tags, you can easily filter the Queries UI by the one-off or deprecated tags and delete them quarterly. You can also look at the "Last Run" column in the Queries list to quickly spot queries that haven't been touched in months.

4. If your team wants version control later, you can save these as .sql files inside Databricks Git Folders (Repos). However, if your main goal is simplicity right now, just stick to the DBSQL Queries UI with strict tagging and folder management.

Re: How to manage SQL queries for business data extraction in Databricks?

balajij8 — Tue, 11 Aug 2026 10:25:36 GMT

The catalog table generally adds maintenance overhead in manual cases as it can drift out of sync with actual queries if managed manually. Databricks capabilities might be enough as you can use Workspace search (searches across query names and content including SQL comments), Folder browsing works fine up to 100-200 queries per folder, System tables automatically tracks who ran what and when and SQL comments contain all the metadata you need

You can start simple and add complexity only if needed.

Start with: Folders + naming convention + SQL comment metadata
Use workspace search when you need to find something
Query system tables periodically to identify unused queries & usage statistics
Add a catalog table or an app that takes the request info & load the catalog table or you can run it via workflow when Workspace search isn't sufficient & you get complex reporting requests

You can skip additional table unless you're managing thousands of queries or need sophisticated reporting/analytics on the requests as the lightweight approach (folders + naming + comments) is usually sufficient for direct cases.

90-day rule is a trigger to highlight unused queries. You can archive it using workflows after a review.

Re: How to manage SQL queries for business data extraction in Databricks?

HILL22 — Tue, 11 Aug 2026 12:03:39 GMT

Thanks for the detailed suggestions! This is very helpful. I think the lightweight approach with domain-based folders, naming conventions, metadata, and an archive process fits our use case very well. I really like the idea of keeping the management simple within Databricks without introducing unnecessary overhead.

Re: How to manage SQL queries for business data extraction in Databricks?

HILL22 — Tue, 11 Aug 2026 12:04:10 GMT

Thanks for the great suggestions! I hadn't considered using Query Tags in this way. Combining tags with folders and naming conventions seems like a practical approach for keeping the queries organized as they grow. Really appreciate the helpful advice!