topic Re: Creating Databricks agent in Data Engineering

Creating Databricks agent

DineshOjha — Mon, 10 Aug 2026 22:30:42 GMT

Hi All,

We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new agent.

We are working on a migration project, moving data from Hive to Databricks, so we want to create a data validation agent which can query both hive and databricks.

Can you please guide us on how to proceed?

Re: Creating Databricks agent

Sumit_7 — Tue, 11 Aug 2026 03:04:08 GMT

@DineshOjha This doc will be useful in proceeding https://docs.databricks.com/aws/en/agents/. Give it a try and let me know.

Re: Creating Databricks agent

DineshOjha — Tue, 18 Aug 2026 20:34:36 GMT

Thank you , we had looked at this but this doesnt help to create an agent from start to finish. Is there any other exercise or example we can look at?

Re: Creating Databricks agent

balajij8 — Wed, 19 Aug 2026 03:14:38 GMT

Dinesh,

You can create agents that are tuned on data from multiple systems. Also, you can use any model from classical ML to Gen AI based on application’s needs.

You can find the step-by-step example here

You can find another example here

Re: Creating Databricks agent

kartikchoudhary — Wed, 19 Aug 2026 13:20:26 GMT

If you're approaching this specifically as a migration validation use case, I'd separate the agent into two layers: the validation tools and the agent reasoning layer.

For example, expose controlled tools for querying Hive and Databricks, then have the agent compare row counts, schemas, null rates, key-level mismatches, and aggregate-level differences. This makes the agent useful for repeatable migration validation rather than just a chatbot over both data sources.

We’ve seen this pattern become particularly useful in enterprise data modernization projects, where the goal is not only to migrate data but also to automate the validation process across the old and new platforms.