https://community.databricks.com/t5/technical-blog/combining-agent-bricks-and-ai-parse-document-for-smarter/ba-p/143801 <H1>Building a Research Paper Curator for Knowledge Assistants</H1> <P><SPAN>Document-heavy workflows such as research analysis, contract review, and support ticket processing, are a natural fit for AI agents. Building them from scratch means orchestrating document parsing, information extraction, vector search, chat interfaces, and likely much more. That’s a lot of plumbing before you get to the interesting part.</SPAN></P> <P><SPAN>Databricks Agent Bricks and the </SPAN><FONT face="courier new,courier"><SPAN>ai_parse_document</SPAN></FONT><SPAN> SQL function eliminate most of that plumbing. In this post, we’ll show how these tools work together by building an application for curating research papers for a knowledge assistant. Using this example, we’ll highlight patterns you can apply to your own agentic applications on Databricks.</SPAN></P> <P><SPAN>What we’ll use:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><FONT face="courier new,courier"><SPAN>ai_parse_document</SPAN></FONT><SPAN> Databricks SQL function for extracting structured text from PDFs</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/generative-ai/agent-bricks/key-info-extraction" target="_blank" rel="noopener"><SPAN>Agent Bricks Key Information Extraction</SPAN></A><SPAN> for pulling structured fields from documents</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/generative-ai/agent-bricks/knowledge-assistant" target="_blank" rel="noopener"><SPAN>Agent Bricks Knowledge Assistant</SPAN></A><SPAN> for RAG-based Q&amp;A</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/aws/en/dev-tools/databricks-apps/" target="_blank" rel="noopener"><SPAN>Databricks Apps</SPAN></A><SPAN> for the application interface</SPAN></LI> </UL> <P><SPAN>This is a high-level structural overview, so we won’t walk through every line of code. Instead, we’ll focus on how these components interact and unlock value together. For the full implementation, </SPAN><A href="https://github.com/databricks-solutions/devrel-examples/tree/main/demo_projects/arxiv" target="_blank" rel="noopener"><SPAN>check out the project on GitHub</SPAN></A><SPAN>.</SPAN></P> <H2><STRONG>The Problem: Knowledge Assistants Need Curation</STRONG></H2> <P><SPAN>Knowledge assistants such as the </SPAN><A href="https://docs.databricks.com/aws/en/generative-ai/agent-bricks/knowledge-assistant" target="_blank" rel="noopener"><SPAN>Databricks Agent Bricks Knowledge Assistant</SPAN></A><SPAN> work best when they have access to focused, relevant content. Overloading them with irrelevant materials can lead to knowledge assistants retrieving only some of the relevant information they need to address a query or, worse, retrieving entirely incorrect information (e.g., </SPAN><A href="https://aclanthology.org/2025.acl-long.131/" target="_blank" rel="noopener"><SPAN>1</SPAN></A><SPAN>, </SPAN><A href="https://arxiv.org/abs/2601.01896" target="_blank" rel="noopener"><SPAN>2</SPAN></A><SPAN>).</SPAN></P> <P><SPAN>Manually pre-screening materials for inclusion in a knowledge assistant can be a cumbersome and time-consuming task, especially when it comes to technical documents like research papers. You may need to search through dozens of pages of text to determine whether a paper warrants inclusion in your knowledge assistant.</SPAN></P> <H2><STRONG>The Solution: A Curation Workflow</STRONG></H2> <P><SPAN>In order to make the screening process as painless as possible, we can make use of several AI features on Databricks to pull out some key details from the research papers and a Databricks app to create a simple UI for flagging papers to include.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_1-1768248344551.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22912i4B027061FFCE537A/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_1-1768248344551.png" alt="DanielLiden_1-1768248344551.png" /></span></P> <P><I><SPAN>The curator app: search arXiv, parse and extract with AI, manage your knowledge base</SPAN></I></P> <P><SPAN>To accomplish this, we will:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Search the arXiv API for papers based on topics and keywords</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Get the text from promising candidates with the </SPAN><SPAN>ai_parse_document</SPAN><SPAN> SQL function</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Extract key details from the candidate papers with Agent Bricks Information Extraction</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Review the extracted fields, which are targeted and relevant to the criteria we’re using to filter papers for inclusion in our Knowledge Assistant. This is much quicker than manual review!</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Add relevant papers to the knowledge assistant.</SPAN></LI> </OL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_0-1768248344550.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22911i5FBDEA9573BA4244/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_0-1768248344550.png" alt="DanielLiden_0-1768248344550.png" /></span></P> <P><EM>Document search, parsing, and review workflow for knowledge assistant curation</EM></P> <H2><STRONG>Setting Up the Agents</STRONG></H2> <P><SPAN>Agent Bricks agents are tailored to specific tasks and data, so we need to initialize the Information Extraction and Knowledge Assistant agents before we can integrate them into our application.</SPAN></P> <P><SPAN>After the agents are configured (via the Databricks UI), we will invoke them via the OpenAI SDK. Agent Bricks endpoints are OpenAI-compatible, so you can use the familiar OpenAI SDK pattern:</SPAN></P> <LI-CODE lang="python">from databricks.sdk import WorkspaceClient ws_client = WorkspaceClient() openai_client = ws_client.serving_endpoints.get_open_ai_client() response = openai_client.chat.completions.create( model="your-agent-endpoint", messages=[{"role": "user", "content": "Your prompt here"}], )</LI-CODE> <P><SPAN>This pattern works for both KIE and Knowledge Assistant endpoints. The </SPAN><FONT face="courier new,courier"><SPAN>WorkspaceClient</SPAN></FONT><SPAN> handles authentication automatically.</SPAN></P> <P><SPAN>Here’s how to set up the Agent Bricks agents.</SPAN></P> <H3><STRONG>Key Information Extraction (KIE)</STRONG></H3> <P><SPAN>The KIE agent extracts structured fields from parsed documents, giving you a quick preview of each paper’s contributions, methodology, and limitations without reading 30 pages. You configure what to extract by defining a JSON schema, then point the agent at your parsed text. The </SPAN><A href="https://docs.databricks.com/aws/en/generative-ai/agent-bricks/key-info-extraction" target="_blank" rel="noopener"><SPAN>full KIE documentation</SPAN></A><SPAN> covers all configuration options.</SPAN></P> <P><SPAN>Preparing the data. Before KIE can extract anything, PDFs need to be converted to text. We started with a golden set of seminal LLM agent papers (ReAct, Reflexion, etc.) and parsed them using the </SPAN><SPAN>ai_parse_document</SPAN><SPAN> SQL function:</SPAN></P> <P><STRONG><FONT face="courier new,courier">SELECT ai_parse_document(content) as parsed</FONT></STRONG><BR /><STRONG><FONT face="courier new,courier">FROM read_files('/Volumes/catalog/schema/volume/paper.pdf')</FONT></STRONG></P> <P><SPAN>The </SPAN><FONT face="courier new,courier"><SPAN>ai_parse_document</SPAN></FONT><SPAN> function handles multi-column layouts, equations, and citations, returning structured JSON with page-by-page content. We stored the extracted text in a </SPAN><FONT face="courier new,courier"><SPAN>parsed_documents</SPAN></FONT><SPAN> table, which becomes the KIE agent’s data source.</SPAN></P> <P><SPAN>Defining the schema. The Agent Bricks UI generates a starter schema based on your data, but you’ll want to replace it with fields relevant to your use case. Switch to JSON schema mode and define what you need. For paper screening, we defined seven fields: </SPAN><FONT face="courier new,courier"><SPAN>title</SPAN></FONT><SPAN>, </SPAN><FONT face="courier new,courier"><SPAN>authors</SPAN></FONT><SPAN>, </SPAN><FONT face="courier new,courier"><SPAN>affiliation</SPAN></FONT><SPAN>, </SPAN><FONT face="courier new,courier">methodology</FONT><SPAN>, </SPAN><FONT face="courier new,courier"><SPAN>contributions</SPAN></FONT><SPAN>, </SPAN><FONT face="courier new,courier"><SPAN>limitations</SPAN></FONT><SPAN>, and </SPAN><FONT face="courier new,courier"><SPAN>topics</SPAN></FONT><SPAN>. Here’s an excerpt:</SPAN></P> <LI-CODE lang="javascript">{ "properties": { ... "methodology": { "type": "string", "description": "Research methods, model architectures, and experimental approaches used." }, "limitations": { "type": "array", "items": {"type": "string"}, "description": "Acknowledged weaknesses, constraints, and areas for improvement." }, ... } } </LI-CODE> <P><SPAN>The </SPAN><SPAN>description</SPAN><SPAN> fields matter: they guide the model’s extraction logic. If you’re getting inconsistent results, refining the description often helps more than adding examples. We also used </SPAN><SPAN>anyOf</SPAN><SPAN> with null for optional fields (like </SPAN><SPAN>affiliation</SPAN><SPAN>) so the model returns </SPAN><SPAN>null</SPAN><SPAN> when information isn’t present rather than hallucinating.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_2-1768248344552.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22913iF7A4F5C4502EB50F/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_2-1768248344552.png" alt="DanielLiden_2-1768248344552.png" /></span></P> <P><I><SPAN>KIE agent configuration showing extraction schema and sample outputs</SPAN></I></P> <H3><STRONG>Knowledge Assistant</STRONG></H3> <P><SPAN>The Knowledge Assistant provides RAG-based Q&amp;A over your documents. Setup is straightforward: point it at a Unity Catalog Volume and deploy. The </SPAN><A href="https://docs.databricks.com/aws/en/generative-ai/agent-bricks/knowledge-assistant" target="_blank" rel="noopener"><SPAN>full Knowledge Assistant documentation</SPAN></A><SPAN> covers additional options like vector search indexes and custom instructions.</SPAN></P> <P><SPAN>Configuring the data source. In the Agent Bricks UI, select Knowledge Assistant and click Build. You’ll need to specify:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Name: We used </SPAN><FONT face="courier new,courier"><SPAN>arxiv-papers</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Knowledge Source: Select Unity Catalog Volume and point it at your PDFs volume (e.g., </SPAN><FONT face="courier new,courier"><SPAN>arxiv_demo.main.pdfs</SPAN></FONT><SPAN>)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Instructions (optional): Guidelines for how the agent should respond</SPAN></LI> </UL> <P><SPAN>Only documents in this volume will be indexed. For the curator application, we set up a separate staging volume for papers under review. documents only move to the KA volume when explicitly approved. This keeps the knowledge base focused.</SPAN></P> <P><SPAN>After deployment, the agent takes a few minutes to sync and index your documents. You can test it immediately via the embedded chat or AI Playground. If you add or remove files from the volume later, click Sync in the UI to update the index.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_3-1768248344553.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22915i6D91346B47B7B480/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_3-1768248344553.png" alt="DanielLiden_3-1768248344553.png" /></span></P> <P><I><SPAN>Knowledge Assistant configured with a UC Volume source</SPAN></I></P> <H2><STRONG>The Curation Workflow</STRONG></H2> <P><SPAN>We built a Databricks App with a four-phase workflow that calls upon these agents:</SPAN></P> <H3><STRONG>Phase 1: Search</STRONG></H3> <P><SPAN>Search arXiv for papers matching your criteria. The UI lets you filter by category, date range, and keywords. You can select papers that look like they </SPAN><I><SPAN>might</SPAN></I><SPAN> be relevant to your interests and worth including in the knowledge assistant.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_4-1768248344553.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22914i2B02CB70DF688C79/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_4-1768248344553.png" alt="DanielLiden_4-1768248344553.png" /></span></P> <P><I><SPAN>Search interface with category filters and date range</SPAN></I></P> <H3><STRONG>Phase 2: Review (Parse + Extract)</STRONG></H3> <P><SPAN>Once you select some papers, you can click the “Parse Papers” button beneath the list. This triggers the following:</SPAN></P> <OL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The selected papers are downloaded and uploaded to a staging volume. Papers in the staging volume are </SPAN><I><SPAN>not</SPAN></I><SPAN> accessible by the Knowledge Assistant.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The </SPAN><FONT face="courier new,courier"><SPAN>ai_parse_document</SPAN></FONT><SPAN> SQL function is used to obtain the text of the papers and save them to a Unity Catalog table.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The Information Extraction agent we defined previously extracts the relevant fields from the papers.</SPAN></LI> </OL> <P><SPAN>Once the parsing and extraction are complete (which might take a few minutes, depending on the number of papers), you can review the key contributions, methods, and limitations of each paper in the review tab. From the review tab, you can select which papers to add to the knowledge assistant.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_5-1768248344554.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22916iFBFBAC8859443706/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_5-1768248344554.png" alt="DanielLiden_5-1768248344554.png" /></span></P> <P><I><SPAN>KIE-extracted insights showing key contributions and methodology</SPAN></I></P> <H3><STRONG>Phase 3: Manage</STRONG></H3> <P><SPAN>The KA Manager shows what’s currently in your knowledge base. You can add or remove documents. Keeping the knowledge assistant up to date with relevant materials helps to keep the results targeted and relevant over time.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_6-1768248344555.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22918iB0A80E28661F93B0/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_6-1768248344555.png" alt="DanielLiden_6-1768248344555.png" /></span></P> <P><I><SPAN>Knowledge Assistant Manager showing curated papers</SPAN></I></P> <H3><STRONG>Phase 4: Chat</STRONG></H3> <P><SPAN>Query your curated knowledge base. Because you’ve been selective about what goes in, responses are focused and relevant.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DanielLiden_7-1768248344555.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/22917i0C766DE5808E14E1/image-size/large?v=v2&amp;px=999" role="button" title="DanielLiden_7-1768248344555.png" alt="DanielLiden_7-1768248344555.png" /></span></P> <P><I><SPAN>Chat interface querying the curated knowledge base</SPAN></I></P> <P><SPAN>There are several other ways you can chat with the knowledge assistant. You can use the Knowledge Assistant UI directly in Databricks by clicking on the “Agents” tab, selecting your knowledge assistant, and using the Knowledge Assistant chat interface. You can also use the Databricks AI Playground by clicking the Playground tab and finding your Knowledge Assistant endpoint in the model selection dropdown. We included the chat interface directly in the curator application primarily for convenience.</SPAN></P> <H2><STRONG>Key Takeaway</STRONG></H2> <P><SPAN>We used a paper curator as our example, but the real takeaway is how quickly you can build document-processing agents on Databricks. Agent Bricks + </SPAN><FONT face="courier new,courier"><SPAN>ai_parse_document</SPAN></FONT><SPAN> handle the hard parts, so you can focus on your use case.</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Knowledge Assistant gives you a Q&amp;A bot over your documents in just a few UI-driven steps, without needing to worry about configuring a vector database or chat interface yourself.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The Information Extraction agent can obtain key details from complex documents and return them in a structure you define. Again, this takes only a few UI-driven steps.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><FONT face="courier new,courier"><SPAN>ai_parse_document</SPAN></FONT><SPAN> makes it easy to extract clean, structured text from documents and excels at complicated documents like research papers that may include tables and figures.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Databricks Apps let you define custom application logic for tying all the pieces together.</SPAN></LI> </UL> <P><SPAN>In this example, we developed a curator app that simplifies the process of adding relevant data to a knowledge assistant. But the broader principles apply across use cases. With Databricks Agent Bricks, creating performant, high-quality agents that integrate seamlessly with your applications and workflows has never been easier, and they’re even more powerful when used together.</SPAN></P> <H2><STRONG>Try It Yourself</STRONG></H2> <P><A href="https://github.com/databricks-solutions/devrel-examples/tree/main/demo_projects/arxiv" target="_blank" rel="noopener"><SPAN>Check out the project on GitHub</SPAN></A><SPAN>. To get started, import the repository to your Databricks workspace and follow the steps in the project’s </SPAN><A href="https://github.com/databricks-solutions/devrel-examples/blob/main/demo_projects/arxiv/Runbook.ipynb" target="_blank" rel="noopener"><SPAN>runbook</SPAN></A><SPAN>. This will guide you through the process of setting up the agents, creating your agents, and deploying the application.</SPAN></P> Mon, 12 Jan 2026 20:58:11 GMT Daniel-Liden 2026-01-12T20:58:11Z https://community.databricks.com/t5/technical-blog/combining-agent-bricks-and-ai-parse-document-for-smarter/ba-p/143801 <P>Document workflows need a lot of plumbing, such as parsing, extraction, vector search, and chat interfaces, before you get meaningful and usable results. Agent Bricks and ai_parse_document handle the hard parts. We built a research paper curator to show how these tools work together, but the patterns apply wherever you're turning documents into decisions.</P> Mon, 12 Jan 2026 20:58:11 GMT https://community.databricks.com/t5/technical-blog/combining-agent-bricks-and-ai-parse-document-for-smarter/ba-p/143801 Daniel-Liden 2026-01-12T20:58:11Z https://community.databricks.com/t5/technical-blog/combining-agent-bricks-and-ai-parse-document-for-smarter/bc-p/144100#M884 <P>this is fire. thanks for sharing</P> Wed, 14 Jan 2026 22:19:33 GMT https://community.databricks.com/t5/technical-blog/combining-agent-bricks-and-ai-parse-document-for-smarter/bc-p/144100#M884 npitts 2026-01-14T22:19:33Z