Deep Dive: Context Graphs

This is part two in a five-part series diving into the Amigo Cognitive Architecture. We’ve already covered Functional Memory – in upcoming posts we will explore Dynamic Behaviors, Actions, and the Agent Core.

Almost all AI agents fall short for real-world healthcare use cases because they can't balance structure with flexibility. They're either too rigid to handle the nuance of real patient cases, or too loose to maintain the clinical rigor that safety demands.

Amigo solves this problem with context graphs – a proprietary architecture that reimagines how to navigate complex conversations. Unlike traditional agent frameworks that rely on either linear decision trees or largely unstructured reasoning, context graphs provide external scaffolding that preserves clinical logic while enabling personalized adaptation. They lay out the broader landscape of a task, its purpose, its structure, and the intricate connections that hold it together. The outcome is agents that can follow vetted protocols as reliably as the best clinicians, while flexing to each patient's unique circumstances.

How do you teach an AI agent to navigate complex clinical conversations at scale?

This was the core question that led us to design the context graph, a blueprint that helps agents navigate a specific problem space by breaking it down into smaller, connected parts. Picture a climber scaling a mountain face. The context graph represents pre-mapped footholds that show the climber (the agent) validated pathways to navigate upward. These footholds present them with multiple safe paths they can choose based on real-time factors like weather conditions or available equipment.

Balancing Structure with Flexibility#

Having a clear structure is crucial – a PCP seeing a patient for their annual visit follows a specific flow that is comprehensive and likely to surface critical issues. But cases are often complex and every patient requires a degree of personalization based on factors like medical history and health literacy. This interplay means the agent must appropriately balance clinically tried-and-true structure with the ability to adapt as needed.

Context graphs solve this by offering a flexible spectrum that clinical teams can calibrate to their specific requirements:

• Strict contexts for critical clinical protocols (e.g., medication instructions, safety procedures)
• Medium-flexibility contexts for clinical guidance (e.g., treatment discussions, care planning)
• Open-ended contexts for patient conversations (e.g., empathetic guidance, building rapport)

Bridging the Token Bottleneck#

Another critical function of the context graph architecture is that it overcomes a significant limitation faced by current AI models that we call the token bottleneck.

Imagine a genius with short-term memory loss who reasons by writing down one word at a time. Each time they write a word, they completely lose their memory and have to reconstruct their reasoning by reading previous words.

This is the challenge faced by today’s LLMs. When models need to “think through” problems step-by-step, they must compress their reasoning into text tokens to express it. One token is emitted and the internal state is effectively reset. The model must then rebuild context from its output.

This causes a significant loss in reasoning that is unacceptable in a high-stakes, high-complexity domain like healthcare. Clinical decision-making involves vast amounts of interconnected information like patient histories, regulatory requirements, and clinical pathways, yet foundation models cannot hold onto the majority of this context.

We designed context graphs to act as external scaffolding for agents to organize and preserve their reasoning. Instead of letting the model lose track of the conversation as it progresses, the context graph holds important details in place so the agent can frame its responses correctly. For example, an intake agent talking to a patient will know at all times what they’ve already covered and where the conversation needs to go next, allowing it to stay on topic and frame its questions appropriately.

Anatomy of a Context Graph#

At its core, a context graph organizes decision points into layered states, enabling an agent to navigate complex behaviors efficiently. Instead of branching in one direction along preset “rails” like a decision tree, a context graph works more like a spiderweb; the agent can travel forward, backward, or laterally along a number of valid pathways based on situational context.

There are multiple types of states, each playing a distinct role in guiding an agent's behavior and managing conversation flow:

• Action states execute tasks or respond to the user within established rules and constraints, guided by the active conversational context
• Decision states determine optimal actions based on real-time data and objectives, simultaneously drawing on memory, knowledge, and reasoning
• Reflection states enable deeper thought and force the agent to carefully consider its reasoning before moving forward
• Recall states explicitly retrieve user memory or past interactions to personalize and improve responses, bringing historical context into play
• Annotation states clarify and segment complex interactions to help the agent keep tabs on key information
• Side-effect states identify points where the agent can interact with external systems or trigger actions outside of its own environment

Amigo’s Agent Engineers work with our partners’ clinical teams to deeply understand the structural topology of the problem the agent is meant to solve, then use these core building blocks to construct the optimal context graph. This state-based architecture also allows the agent’s reasoning to be broken down into clear, traceable steps that make it possible to audit conversations with extreme visibility.

Layering in Functional Memory#

Context graphs and functional memory work as complementary systems. While the context graph provides the structural roadmap for clinical conversations, the memory system ensures the agent navigates that roadmap with the right patient knowledge at the right time.

The agent's user model stays active throughout navigation, providing continuous access to the complete patient picture as it moves through the context graph. This enables the agent to make informed decisions at each junction and respond appropriately based on the patient's clinical profile.

Depending on what the context graph dictates, the agent can also dig deep to perform memory expansion to retrieve specific information missing from the high-level user model. This can happen either implicitly (when the agent determines it has insufficient information to execute on a task demanded by the context graph) or explicitly through a recall state that forces deliberate historical recontextualization.

Together, these systems create a feedback loop: functional memory ensures the agent navigates the context graph with the right contextual framing, which in turn helps the agent access and update its memory more effectively.

Structured Intelligence for Personalized Care#

Like a seasoned clinician who follows established protocols while adapting to each patient's unique circumstances, Amigo's context graph architecture provides the structural foundation for intelligent clinical reasoning. By organizing complex healthcare conversations into interconnected states rather than rigid decision trees, these graphs enable agents to maintain clinical rigor while preserving the flexibility essential for personalized patient care.

Working seamlessly with the functional memory system along with the rest of the Amigo Cognitive Architecture, context graphs ensure that every interaction is both clinically sound and contextually appropriate, allowing healthcare organizations to scale expert-level services across each patient while maintaining the nuanced decision-making that defines quality care.

Interested in how Amigo applies context graphs to build more intelligent clinical agents? Book a call with us today.