Building an AI Powered Autonomous Data Reliability Platform using Databricks & Gemini LLM

VaishnaviSL — Sat, 30 May 2026 17:37:03 GMT

What if a data pipeline could explain why, it failed instead of just saying it failed? 👀

While learning Databricks and exploring Data Engineering, I built an AI Powered Autonomous Data Reliability Platform on Databricks Free Edition using:

🔹 PySpark
🔹 Delta Lake
🔹 Databricks Workflows & Dashboards
🔹 Metadata-driven validation framework
🔹 Gemini LLM integration for AI-powered root cause analysis

The platform dynamically validates large-scale data, detects anomalies, monitors pipeline quality, and generates intelligent remediation insights using Generative AI.

One of my favorite parts of this project was integrating Gemini LLM to transform traditional monitoring into an intelligent observability system 🚀

This project helped me learn:
- workflow orchestration
- scalable validation design
- AI integration in data engineering
- observability concepts
- Medallion Architecture using Databricks

Would love to hear your thoughts and feedback from the community!

GitHub Repository:
Som-115 (vaishnavi)

Demo Video:
https://drive.google.com/file/d/1-7s-idbJmSRdjPlSTPAy2tEsW_4mS-AQ/view?usp=sharing

#Databricks #DAIS2026 #DataEngineering #GenerativeAI #PySpark #DeltaLake #AI #LLM #DataObservability

topic Building an AI Powered Autonomous Data Reliability Platform using Databricks & Gemini LLM in Data Engineering

Building an AI Powered Autonomous Data Reliability Platform using Databricks & Gemini LLM