LakeOps - A Lakehouse Operation Observability App

[ad3ad3]

The Problem LakeOps Solves

Data teams focused on cloud-based analytics solutions often experience operational bottlenecks. Engineers spend countless hours writing SQL joins across system logs monitoring and reporting pipeline failures, while engineering managers struggle to get visible insights into compute costs and SLA performance. Some teams solve this through dashboards with static KPI reporting, but this often limits how much insight could be drawn from the data. LakeOps addresses this by providing an intelligent, natural-language interface for Lakehouse operation observability.

Target Audience and Dual-Persona Design

LakeOps is designed with a context-aware toggle to serve two distinct operational roles:

• Data Engineers: Can ask highly technical debugging questions to retrieve specific error traces, execution times, and job logs for root-cause analysis.
• Engineering Managers: Can request high-level summaries regarding DBU consumption, overall pipeline costs, and business impact.

Application architecture and data flow

LakeOps utilizes a secure architecture deployed directly within the Databricks Data Intelligence Platform. The end-to-end data flow is visualized below:

  Data Layer: Following the Medallion architecture, the pipeline processes data through three layers, all stored in Delta Lake and strictly governed by Unity Catalog:

• Bronze: Raw logs and system tables are used as the bronze layer. Since this project was developed using the Databricks Free Edition workspace, synthetic data was ingested into the tables in this layer.
• Silver: Transforms and cleans the raw data from the bronze layer into parsed pipeline execution records and technical error traces as required.
• Gold: Aggregates the silver data into final business-level metrics, such as total DBU consumption, costs, and SLA performance.

Authentication Flow: The Streamlit application is hosted as a serverless, containerized Databricks App. It utilizes On-Behalf-Of (OBO) authentication, meaning the app's Service Principal is entirely locked out of the data. Instead, it uses the logged-in user's OAuth token, ensuring all queries execute securely under their exact Unity Catalog privileges.

Logic & Processing with Genie Agent: When a user submits a prompt, the app injects the selected persona context (Data Engineer or Engineering Manager). The Databricks Genie Agent processes this combined context, acting as the semantic engine to translate the natural language request into optimized SQL against the permitted tables.

Presentation Layer: The app's custom logic intercepts the returned SQL and data arrays from the Databricks SDK. It then intelligently routes the output to render dynamic, interactive Plotly tabs (Bar, Line, Pie) or raw DataFrames based on the structure of the returned data.

What can users ask the Genie Agent

Since LakeOps is context-aware, queries are inherently tailored to the user's operational role. The application intelligently guides the agent to fetch the appropriate level of detail:

• Engineering Managers can ask business-level, aggregate questions such as:
• "What was our total DBU spend by pipeline this month?"
• "Which pipeline experienced the most SLA breaches last week, and what was the associated compute cost?"
• Data Engineers can dive directly into technical, root-cause investigations by asking:
• "What is the success rate for all pipelines in the last month?"
• "What is the average execution latency for the Gold aggregation job over the past seven days?"

How Genie powers the app's main experience

When a user submits a question, the Streamlit app concatenates the persona-specific instructions and forwards the prompt to Genie. Genie leverages its deep understanding of the underlying table metadata and Unity Catalog relationships to generate optimized SQL queries on the fly. LakeOps then extracts this SQL from the Genie API payload, executes it synchronously against a serverless SQL Warehouse, and uses a custom Python heuristic to dynamically render the results as interactive Plotly charts or raw data tables. Without Genie's real-time text-to-SQL translation and semantic awareness, this dynamic, self-serve observability experience would be impossible.

What I learned while building and testing the app

Building and deploying LakeOps end-to-end within the Databricks Data Intelligence Platform provided several key engineering takeaways:

• Curating Genie Data Models for Accuracy: Grounding Genie in accurate metadata, logical table joins, and clear instructions is essential to prevent hallucinations. Curating focused tables across the Medallion layers ensured Genie consistently chose the correct tables depending on the context of the incoming question.
• Benchmarking Genie Agent Queries: Conducted query benchmarking to establish baseline performance and response accuracy. This testing phase was vital to verify that Genie consistently translated natural language into the most optimized SQL, and to ensure acceptable latency when querying across the varying complexities of the Medallion layers.
• Mastering On-Behalf-Of (OBO) Authentication: Implementing user authorization via the OBO model proved critical for enterprise security. It taught the importance of properly configuring OAuth scopes (genie, workspace, SQL) so the app can securely execute queries under the end-user's identity without relying on overly privileged Service Principals.
• Building Resilient API Parsers: Handling SDK payload structures required robust dictionary-based parsing to extract text attachments and generated SQL statements dynamically across varying API responses.
• End-to-End Serverless Integration: Deploying a containerized Streamlit Databricks App directly where the data lives highlighted the efficiency of the serverless compute plane. Coordinating the workflow—from the UI, through the Genie Agent, across the serverless SQL Warehouse, and back to dynamic Plotly visualizations—demonstrated how seamlessly Databricks components integrate into a unified operational tool.

Final Thoughts and Future Roadmap

While LakeOps successfully demonstrates the power of natural-language observability, the current application is a prototype. Time constraints limited the initial scope, but the following enhancements would significantly elevate both the app's capability and the overall user experience:

1. General User Experience Improvements

• Automated Data Refresh Pipelines: Implementing a scheduled daily pipeline orchestration job to keep data continuously flowing from the Bronze to Gold layers, ensuring that LakeOps always surfaces fresh operational metrics from the previous workday.
• Query Caching: Implementing caching mechanisms (such as Streamlit's @ST.cache_data) for repetitive, high-level SLA queries to ensure instant load times and minimize SQL Warehouse compute costs.
• Proactive Daily Health Checks: Adding automated routines that execute critical SLA and cost queries upon launch, presenting users with a summarized morning health briefing before they even type a prompt.
• Follow-Up Question Suggestions: Utilizing lightweight LLM calls to generate contextual, clickable follow-up questions beneath rendered charts, driving continuous data exploration.

1. RAG Integration for Autonomous Remediation To transition LakeOps from a purely diagnostic tool into an active remediation assistant, the architecture could be expanded with Retrieval-Augmented Generation (RAG):

• Cross-Platform Documentation Search: Indexing official documentation for the core technologies in the team's data stack—such as Databricks and Azure Data Factory. If an engineer encounters an obscure pipeline exception, the RAG model could surface the exact syntax fix or known workaround without requiring them to leave the app.
• Historical Long-Term Memory: Ingesting past Jira tickets, Slack support threads, and incident post-mortems into a Databricks Vector Search index. This would grant the application long-term memory, instantly providing engineers with historical context on recurring pipeline failures and their exact resolution paths.

This development was done using Google Gemini and Genie Code as coding assistant.

#Genie-PoweredAppChallenge