I have been thinking a lot about something Ali Ghodsi (CEO - Databricks) said recently. In a Bloomberg interview, he stated: "We already have artificial general intelligence. We don't need AI to get smarter. It is just lacking context."
After years of building data pipelines and AI systems on Databricks, I can tell you this is the most accurate thing anyone has said about where we are right now.
Every week there is a new model. Better benchmarks. More parameters. Faster inference. But when you are sitting in front of a real business problem, none of that matters if the model does not understand what it is looking at.
I will give you a simple example.
A grocery store has 3,000 products. An AI model can tell you "stock is low, reorder." Great. But should you reorder? Maybe that product has been sitting on the shelf for two weeks and nobody is buying it. Maybe the supplier takes 4 days and the product expires in 3. Maybe a holiday weekend is coming and demand is about to triple.
The model cannot know any of that on its own. That knowledge comes from the data layer underneath it. The Bronze to Silver to Gold pipeline. The business logic encoded in transformations. The risk scores and velocity calculations and seasonal patterns that turn raw numbers into meaning.
That is context. And data engineers are the ones who build it.
I have been working on this idea across three projects recently, all on Databricks.
The first one is Grocery Data Intelligence. It takes raw inventory data through a full medallion pipeline and adds layers of context at each step: days of supply, stockout risk, waste risk, reorder priority. By the time the data reaches the AI reasoning layer, it is not just asking "is stock low." It is asking "given this store, this product, this history, this season, and this supplier lead time, what is the smartest action to take right now." The app is live at grocerydataintelligence.com and I submitted it for the DAIS 2026 Community Virtual Contest.
The second one is Future of Movie Discovery. When someone says "I want something relaxing after a long day," a traditional recommendation engine has no idea what to do with that. It knows your watch history. It knows what is popular. But it does not understand mood. I used embedding models on a Databricks lakehouse to give the AI the context of how a human actually feels. The model is standard. The context pipeline is what makes it work. This project received an Honorable Mention in the 2025 Free Edition Hackathon.
The third one is around payment decisions. Should this customer be allowed to pay on delivery? The answer depends on dozens of behavioral signals: account age, transaction history, device, location, time of day, recent failures. I built a system that creates a real time behavioral profile so the AI decision engine has full context before restricting or allowing a payment method.
Three very different domains. Same pattern every time.
The AI is not the hard part. Getting the right data to the right place in the right shape at the right time with the right meaning attached to it. That is the hard part.
And that is exactly what the lakehouse architecture is designed for. Bronze captures the raw world. Silver cleans and conforms it. Gold encodes business meaning. And on top of that, Genie, AI/BI, agents, and apps can finally do something useful because they have context.
I have been in the Databricks community since 2024 and the one thing I keep seeing in the forums is that the questions people ask are almost never about model accuracy. They are about data quality. Pipeline reliability. Schema evolution. Partitioning strategy. Merge performance. Unity Catalog governance.
Those are all context problems. And solving them is what makes AI actually work in production.
So here is my question to this community.
How are you building context into your pipelines? What does it look like in your day to day work? Are you using Gold layer views, semantic layers, UC Metric Views, or something else to give your AI systems the meaning they need?
Would love to hear what is working for you.
