Data Engineering 3.0 - From Trusted Data Products to Context Layers

balajij8 — Wed, 02 Sep 2026 13:13:51 GMT

Data Engineering 1.0 was largely about integrating, moving, consolidating massive data to support enterprise decisions. Enterprises built ETL pipelines, data warehouses, data lakes and orchestration platforms to bring data together reliably. Data Engineering 2.0 was about making the move to Cloud Lakehouse and making the data trustworthy and reusable. This created a much more mature enterprise data estate. But the arrival of generative and agentic AI changes the question again. An enterprise may now have thousands of tables, hundreds of data products, semantic models, business glossaries, APIs, documents and real-time data. An agent can potentially access much of this information. But access does not mean understanding. An agent investigating a business problem needs to know which information is relevant, how different entities are related, which definitions are authoritative, which rules apply, what happened previously, what is happening now, and what evidence supports a conclusion.

Enterprises may already have all of this information. The problem is that it is distributed across the data estate. This is the context gap - the distance between information that exists in the enterprise and the context an AI system needs to reliably perform a task.

The Agentic Shift

Traditional analytics was designed primarily to help users make vital decisions. A dashboard presents information. A business user interprets it. The user makes the decision. Machine learning automated parts of this process by using data to generate predictions or classifications. Agentic AI introduces a different operating model. An agent can retrieve information, reason over it, use tools, interact with systems and potentially take action. This means the data platform is no longer serving only a human consumer or an analytical model. It is increasingly serving an intelligent system that needs to understand the enterprise before it can act within it. This changes the role of data engineering.

The question is no longer only How do we make trusted data available? It becomes How do we make trusted enterprise information understandable and usable by an AI system for a specific decision? That is where I see the emergence of Data Engineering 3.0.

Data Engineering 3.0

Data Engineering 3.0 is the next evolution of the data engineering discipline. It is not a replacement for Data Engineering 2.0. It builds on it. The important shift is the unit of value. In Data Engineering 1.0, the question was whether data could be moved reliably at scale, processed and consumed in a robust manner. In Data Engineering 2.0, the question became whether data could be trusted and reused in accordance with governance and usage at scale via data products. In Data Engineering 3.0, the question becomes whether an intelligent system has the right context to perform a task reliably. We are moving from engineering data for consumption to increasingly engineering the information environment in which AI operates.

A Trusted Data Product is not the same as Context

Consider a bank that has already built well-governed data products. It may have a Customer data product containing customer information. It may have a Transaction data product containing transactions. It may have a Merchant data product. It may have a Device data product. It may have a Fraud data product containing historical fraud cases. Each of these can be high-quality, governed and independently reusable.

Now imagine an agent is asked Investigate this transaction and determine whether it should be escalated for fraud investigation. The agent does not simply need the Transaction data product. It may need to understand the customer associated with the transaction, the account involved, the merchant, the device used, recent transaction behaviour, previous fraud cases, current fraud indicators and the policies that determine when a transaction should be escalated. The data already exists. But the decision requires those pieces of information to be brought together with their meaning and relationships intact. That assembled information is the context for the decision. A Data Product provides trusted information. A Context Layer makes trusted information relevant to a particular task.

The Context Layer

An AI Context Layer is a decision-oriented capability that brings together the information an AI system needs to perform a specific task. It can include structured data, semantic definitions, relationships between entities, historical information, real-time signals, business policies and supporting evidence. It does not necessarily mean creating another physical copy of the enterprise data. In many cases, it can be a governed layer that composes information from existing data products and other enterprise systems when an agent needs it.

For the fraud investigation example, the context will include

Customer information and history
The transactions and accounts involved
Merchant characteristics
Device relationships
Recent transaction patterns
Previous fraud cases
Current fraud signals
Applicable fraud policies
Evidence supporting the recommendation

The important point is that the agent does not need the entire enterprise. It needs the right context for the decision it is making.

Context is more than Retrieval

The first instinct when discussing AI context is often to think about RAG. RAG is an important capability. It allows an AI system to retrieve relevant information before generating a response. But enterprise decision-making requires more than retrieving documents. An agent may need to understand that a particular transaction belongs to an account, that the account belongs to a customer, that the transaction was performed using a device associated with several accounts, and that the merchant has a particular risk classification. That is not simply a document retrieval problem. It is a problem involving entities, relationships, semantics, history, current state and rules.

Context may come from structured data, documents, knowledge graph, APIs or operational systems or real-time events. The Context Layer is therefore not a specific technology. It is an architectural capability for composing the information required by an intelligent system.

Where Ontology fits

An ontology can make business concepts and relationships explicit. For example, in a banking space, the enterprise may define relationships between Customer, Account, Transaction, Device and Merchant. That semantic structure helps an AI system understand that these are not independent datasets. It understands the business relationships between them. Ontology therefore provides part of the semantic foundation for a Context Layer. Ontology and context are not the same thing. Ontology answers questions such as

What are the important business concepts? , How are they related? , What does a concept mean? , The Context Layer answers a different question. What of this information and meaning does the agent need for this particular task? This distinction is important because it prevents the Context Layer from becoming another attempt to model the entire enterprise upfront.

From Data Quality to Decision Readiness

Data Engineering 2.0 made data quality a fundamental engineering discipline. We ask whether data is accurate, complete, fresh, consistent and governed. Data Engineering 3.0 introduces another dimension. Is the context sufficient for the decision? Consider the fraud example again. The Customer data, Transaction data, Merchant data and Fraud data may be accurate. Yet the agent could still make a poor decision if an important signal is missing from the context. This is why Data Quality and Context Quality are different problems. A data product can be high quality while the decision context built from several data products is incomplete. This suggests that data engineering will increasingly need to think about decision readiness.

Does the agent have the right information? Is it sufficiently fresh?, Are the relationships correct?, Are the business definitions consistent?, Are the relevant policies available? andCan the agent identify the evidence behind its conclusion? These questions sit on top of traditional data quality rather than replacing it.

Context Layer should be driven by the Decision

There is a natural temptation to approach this as another enterprise-wide architecture exercise. An organization may say We need to build an Enterprise Context Layer. The next step becomes an attempt to model every entity, relationship, rule, document and event across the organization. That is where the initiative can quickly become too large. The better starting point is the decision. Check Which agent task would benefit most from better enterprise context? Then define the minimum context required for that task. For a fraud investigation agent, it means starting with Customer, Account, Transaction, Merchant and Device along with the relationships, historical information and policies required for the investigation. The first objective is not to model the bank. It is to make one important decision more reliable. This is a very different way of approaching enterprise semantics.

Context should evolve with the Agent

The first version of a Context Layer will rarely be complete. And it does not need to be. Start with a defined task. Identify the information required. Build the context. Test the agent. Observe where it lacks information or misunderstands meaning. Then extend the context. This creates a practical feedback loop between the agent and the data platform. The agent becomes a source of evidence about where the enterprise’s existing information environment is insufficient for intelligent decision-making. That evidence can drive the next iteration of the data product or semantic layer. Over time, the Context Layer can expand from one decision to multiple decisions and from one domain to connected domains. The goal is not to build the biggest context layer. It is to build the most useful context for the decisions that matter.

What changes for Data Engineering?

Data Engineering 3.0 does not mean data engineers stop building pipelines. The fundamentals remain critical. Data still needs to be ingested, transformed, stored, governed, monitored and served. But the scope of the discipline expands. Data engineers will increasingly need to understand how enterprise information is represented for intelligent systems. That includes areas such as Semantic modeling, Ontology, Knowledge graphs, Entity resolution, Real-time data, Context retrieval, Agent tools, Data and AI governance, Context evaluation and Decision lineage.

The data engineer’s responsibility increasingly moves from Can we deliver the data? to Can an intelligent system understand and use the data correctly in the context of the decision?. That is a significant evolution in the role. The investments made during Data Engineering 2.0 become the foundation for Data Engineering 3.0. Trusted data products provide reliable information. Data quality provides confidence in that information. Governance provides controls. Lineage provides provenance. Semantic models provide business meaning. Real-time platforms provide current state. Ontologies and Knowledge Graphs can provide explicit relationships.

The next step is to bring these capabilities together around the decisions that AI agents need to perform. Context Layer is the new layer built via DE 3.0 on top of the Trusted Data Foundation enterprises have already created in DE 2.0.

topic Data Engineering 3.0 - From Trusted Data Products to Context Layers in Community Articles