Creating Databricks agent

[DineshOjha]

Hi All,

We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new agent.

We are working on a migration project, moving data from Hive to Databricks, so we want to create a data validation agent which can query both hive and databricks.

Can you please guide us on how to proceed?

[Sumit_7]

@DineshOjha This doc will be useful in proceeding https://docs.databricks.com/aws/en/agents/. Give it a try and let me know.

[DineshOjha]

Thank you , we had looked at this but this doesnt help to create an agent from start to finish. Is there any other exercise or example we can look at?

[balajij8]

Dinesh,

You can create agents that are tuned on data from multiple systems. Also, you can use any model from classical ML to Gen AI based on application’s needs.

You can find the step-by-step example here

You can find another example here

[kartikchoudhary]

If you're approaching this specifically as a migration validation use case, I'd separate the agent into two layers: the validation tools and the agent reasoning layer.

For example, expose controlled tools for querying Hive and Databricks, then have the agent compare row counts, schemas, null rates, key-level mismatches, and aggregate-level differences. This makes the agent useful for repeatable migration validation rather than just a chatbot over both data sources.

We’ve seen this pattern become particularly useful in enterprise data modernization projects, where the goal is not only to migrate data but also to automate the validation process across the old and new platforms.