Hi everyone! 👋
I am thrilled to share my final submission for Databricks AppKit Quest 5! I took the initial starter template and evolved it into a full-fledged, serverless Intelligent Knowledge-Base powered by Databricks Apps, Lakebase, and Foundation Model Serving.
Here is my project: 🔗 Live Deployed App: https://my-databricks-app-7474654586328503.aws.databricksapps.com/lakebase 🎥 YouTube Demo Video: https://youtu.be/n8nmNVhKEdg?si=leKi5vBQKmeYzBzJ
🌟 Going the Extra Mile (Training Voucher Eligibility)
To qualify for the Databricks training vouchers, I wanted to really push the boundaries of what this app could do. I implemented five significant custom features beyond the basic requirements:
Dual-Pane Markdown Engine: A fully functional Markdown editor with a live HTML preview tab to format notes professionally.
Focus Time Tracker: An integrated session stopwatch that tracks your active study/work time on a specific note and persists it directly to Lakebase PostgreSQL.
Bidirectional Note Linking: A cross-referencing system that allows users to link related notes together, creating a navigable knowledge graph.
Tokenized RAG Assistant: An AI agent that parses natural language questions (dropping stop words), searches Lakebase in real-time, summarizes insights, and provides clickable source citations.
Advanced Tagging & Priority: Visual priority badges and a dynamic tagging system for better organization.
🤖 Agents, Prompts & How I Built It
The Agent I Used: I used Gemini as my primary coding assistant and thought partner throughout this quest to iteratively build, debug, and deploy the application.
My Prompting Strategy: Instead of asking for everything at once, I used an Iterative Specification Prompting approach:
Template Injection: I fed the agent the DevHub RAG_TEMPLATE.md and told it: "Base the AI Assistant RAG on this template, and update my PLAN.md to reflect this new architecture."
Modular Planning: I prompted the agent to break the build into three distinct plans: 01-database-and-schema.md, 02-rag-assistant-service.md, and 03-frontend-ux.md.
Targeted Debugging Prompts: When I hit roadblocks, I used specific diagnostic prompts like: "My app fails to deploy with a '42501: must be owner of schema app' error in the SQL Editor. How do I fix the Service Principal permissions?" and "The RAG agent is doing an exact substring match and failing on natural language. Write a script to patch the backend SQL query to tokenize keywords instead."
How I Achieved the Tasks:
Schema Evolution: I first expanded the basic Lakebase schema to include tables for notes, tags, note_tags, and note_links with cascading deletes.
Frontend UI/UX: I completely redesigned the React frontend, adding the Markdown toggle (marked.parse), the interactive timer, and a dedicated RAG drawer.
Overcoming RAG Limitations: The biggest challenge was making the RAG agent understand conversational questions. I patched the backend to filter out stop-words (like "what", "is", "about") and dynamically build a parameterized ILIKE SQL query for the remaining tokens.
Deployment: After clearing up the local vs. Service Principal ownership conflicts in the Databricks SQL Editor by dropping the legacy tables, I successfully pushed everything to production using databricks apps deploy.
A huge thank you to the Databricks and AngelHack teams for putting together this incredible AppQuest. It was a fantastic hands-on experience with Lakehouse architecture!
#Databricks #DatabricksAppKit #GenerativeAI #Lakehouse #RAG #Serverless
