In Layer 1 we built the AI Gateway . Every request gets authenticated, authorized, and policy-checked before it touches a model.
In Layer 2 we built the Agent Planner. It broke our churn analysis query into three sub-tasks and called a different tool for each one.
One of those tool calls went to the Knowledge Layer. This is Layer 3. And it is the layer most enterprise AI implementations get wrong.
The problem with models and enterprise knowledge
Large language models (LLM) know a great deal about the world. They know nothing about your company.
LLM cannot tell you which of your customers are at risk this quarter.
They cannot reference your Q2 product adoption metrics. They cannot explain what your internal churn playbook recommends. That information was not in their training data.
The obvious fix is to fine-tune the model on your internal documents. The problem is that fine-tuning is expensive, slow, and the moment your documents change the model is out of date again. The right answer is retrieval.
Instead of baking knowledge into the model, you retrieve it at query time. Find the most relevant documents, pass them to the model as context, and instruct it to answer from that evidence rather than from training data.
Why grounding matters
In 2026, trust must be earned through architecture, not assumed from the model. The architecture that earns trust in production combines four things:
This is the current best practice consensus across enterprise AI deployments in 2026, as described by ClarityArc in their research on AI hallucination and grounding in enterprise knowledge systems.
The important thing to understand is what grounding does and does not do. It reduces the surface area of possible fabrication. It does not eliminate it. A grounded model can still misrepresent the evidence it was given. That is why citation matters as much as retrieval. Every chunk the Knowledge Layer returns carries its source document and page number. When the model generates its answer, the evidence is traceable back to a specific page of a specific document. In an enterprise audit, that traceability is the difference between a response you can defend and one you cannot.
What the retrieval pipeline looks like
The Knowledge Layer runs seven steps every time it is called.
The query arrives from the Agent Planner as a tool call. The same churn analysis request that passed through the Gateway and was decomposed by the Planner now reaches the Knowledge Layer as a specific sub-task: retrieve context about product adoption signals and customer engagement patterns.
The query is converted into a vector using the all-MiniLM-L6-v2 embedding model. This captures the semantic meaning of the query, not just the keywords but the intent behind it. A query about product adoption signals will retrieve a chunk about feature utilisation thresholds even if those exact words do not appear in the query.
The embedding is compared against every chunk in the ChromaDB collection using cosine similarity. The top chunks surface, scored and ordered by relevance. In our demo run, the Q2 Churn Analysis Report scored 77 percent and the Product Adoption Playbook scored 71 percent.
Chunks below a similarity threshold of 0.5 are removed. Uncertain retrieval is worse than no retrieval because it gives the model something plausible but weak to reason from.
Each retained chunk is tagged with its source document and page number. These citations travel with the context all the way to the model. When the model generates its answer, every factual claim is traceable back to a specific page of a specific document.
The grounded context, three chunks and approximately 408 tokens, retrieved in 218 milliseconds, is handed back to the Planner for synthesis with the SQL and CRM results.
What the demo shows
Running the churn analysis query through Layer 3 in NexusIQ returns three chunks.
The first tells the model that EMEA churn increased 14 percent in Q2 and that product adoption below 40 percent correlates with 85 percent churn probability. The second gives the specific at-risk regions and recommended actions. The third provides the revenue impact correlation across regions.
The model did not know any of this. It retrieved it from verified enterprise documents, with page-level citations, in under 250 milliseconds.
The one architectural decision that matters most
Chunking strategy is the most consequential technical decision in any RAG implementation. Chunk too small and you lose context. Chunk too large and you dilute relevance. In NexusIQ we chunk at the section level, roughly 150 to 400 tokens per chunk, small enough to be specific and large enough to carry meaning.
The similarity threshold is the second decision. Setting it too low lets irrelevant chunks through. Setting it too high means you miss relevant content. 0.5 cosine similarity is the starting point, tuned per deployment based on document type and query patterns.
The connection to the AI Gateway and Agent Planner
The Knowledge Layer does not run independently.
It runs because the AI Gateway approved the knowledge tool for this role and domain. An Analyst querying HR data would never reach the Knowledge Layer. The Gateway blocks it before the Planner even runs.
It runs because the Agent Planner decided this query needed document context alongside structured data. It called the Knowledge Layer as one of three parallel tool calls.
Three layers. One query. Each layer doing exactly one job.
Live Demo
What comes next
Layer 4 is Agent Memory.
We have now run the same churn analysis query three times across three layers. In a real enterprise system the agent should remember what it retrieved, what the SQL returned, and what it recommended the first time. That is the difference between a stateless tool and an agent that learns from its interactions.
Layer 4 shows how.
Series Reference:
Episode 1 - AI Gateway
Episode 2 - Agent Planner
Enterprise Reference:
ClarityArc Consulting. AI Hallucination and Grounding: How Citation Actually Works in Enterprise Knowledge Systems. May 2026. ClarityArc Insights