Hi everyone! 👋

I am thrilled to share my final submission for Databricks AppKit Quest 5! I took the initial starter template and evolved it into a full-fledged, serverless Intelligent Knowledge-Base powered by Databricks Apps, Lakebase, and Foundation Model Serving.

Here is my project: 🔗 Live Deployed App: https://my-databricks-app-7474654586328503.aws.databricksapps.com/lakebase 🎥 YouTube Demo Video: https://youtu.be/n8nmNVhKEdg?si=leKi5vBQKmeYzBzJ

🌟 Going the Extra Mile (Training Voucher Eligibility)

To qualify for the Databricks training vouchers, I wanted to really push the boundaries of what this app could do. I implemented five significant custom features beyond the basic requirements:

1. Dual-Pane Markdown Engine: A fully functional Markdown editor with a live HTML preview tab to format notes professionally.

2. Focus Time Tracker: An integrated session stopwatch that tracks your active study/work time on a specific note and persists it directly to Lakebase PostgreSQL.

3. Bidirectional Note Linking: A cross-referencing system that allows users to link related notes together, creating a navigable knowledge graph.

4. Tokenized RAG Assistant: An AI agent that parses natural language questions (dropping stop words), searches Lakebase in real-time, summarizes insights, and provides clickable source citations.

5. Advanced Tagging & Priority: Visual priority badges and a dynamic tagging system for better organization.

🤖 Agents, Prompts & How I Built It

The Agent I Used: I used Gemini as my primary coding assistant and thought partner throughout this quest to iteratively build, debug, and deploy the application.

My Prompting Strategy: Instead of asking for everything at once, I used an Iterative Specification Prompting approach:

• Template Injection: I fed the agent the DevHub RAG_TEMPLATE.md and told it: "Base the AI Assistant RAG on this template, and update my PLAN.md to reflect this new architecture."

• Modular Planning: I prompted the agent to break the build into three distinct plans: 01-database-and-schema.md, 02-rag-assistant-service.md, and 03-frontend-ux.md.

• Targeted Debugging Prompts: When I hit roadblocks, I used specific diagnostic prompts like: "My app fails to deploy with a '42501: must be owner of schema app' error in the SQL Editor. How do I fix the Service Principal permissions?" and "The RAG agent is doing an exact substring match and failing on natural language. Write a script to patch the backend SQL query to tokenize keywords instead."

How I Achieved the Tasks:

1. Schema Evolution: I first expanded the basic Lakebase schema to include tables for notes, tags, note_tags, and note_links with cascading deletes.

2. Frontend UI/UX: I completely redesigned the React frontend, adding the Markdown toggle (marked.parse), the interactive timer, and a dedicated RAG drawer.

3. Overcoming RAG Limitations: The biggest challenge was making the RAG agent understand conversational questions. I patched the backend to filter out stop-words (like "what", "is", "about") and dynamically build a parameterized ILIKE SQL query for the remaining tokens.

4. Deployment: After clearing up the local vs. Service Principal ownership conflicts in the Databricks SQL Editor by dropping the legacy tables, I successfully pushed everything to production using databricks apps deploy.

A huge thank you to the Databricks and AngelHack teams for putting together this incredible AppQuest. It was a fantastic hands-on experience with Lakehouse architecture!

#Databricks #DatabricksAppKit #GenerativeAI #Lakehouse #RAG #Serverless