Agent architecture: why the gold is in your chats
AI

🧠 Agent architecture: why the gold is in your chats

How to build a personal agent with persistent memory and interchangeable reasoning engines

I took a long pause. I stepped away on June 1 and returned last week. Almost six weeks without writing, without dashboards, without the feeds of a digital world that accelerates a little more each day. I disconnected to reconnect with the people who matter - family, friends, and people I had just met - and with experiences that do not fit on a screen. It was a necessary pause to gain perspective and not forget what is fundamental, what is essential: becoming better human beings, day by day. Today I return to the blog with an architectural decision I had postponed for months and that the pause helped me resolve.

The project is a personal agent, one that inherits my reasoning patterns and accompanies me in real cognitive work, not as a conversational assistant but as a digital colleague with persistent memory. The question I had not resolved was not whether to build it, but how.

The architecture answered the question of where the agent should live. The pause helped me see that its most valuable component, the evidence of my own criteria, already existed inside my conversations.

The agent layer and its engines

My agent was not born with Hermes. It was born in OpenClaw. For months I ran a customized OpenClaw deployment, which I called Nova, on a modest server in London, with a reverse proxy, Zero Trust access, a WhatsApp bridge, and a secrets manager. It worked. And when something works, moving it has to be justified by more than novelty.

In April, I evaluated Hermes for the first time. The Nous Research team had released it on February 25, 2026, and it was growing fast, with a different memory architecture and skills generated from the agent’s own operational experience. I analyzed and compared the two frameworks, then decided not to migrate. Nova was stable, and changing frameworks out of curiosity is exactly the kind of move I do not make. I set up Hermes on an isolated droplet to test it without touching production and kept operating on OpenClaw.

What eventually moved me was not a trend or a headline. It was a recurring bug, the kind of operational friction I had already seen when an OpenClaw update went wrong. In May, a problem that kept returning led me to compare the two frameworks again, this time with fresh data and a concrete reason to reconsider. Hermes addressed at the architectural level problems that in OpenClaw depended on chained patches: memory with native search, skills generated from experience, and an official migration path. That was when I migrated, in phases, with the decision backed by evidence rather than enthusiasm.

The broader ecosystem reinforced the same lesson. Peter Steinberger, creator of OpenClaw, joined OpenAI in February. On July 8, OpenClaw announced its transition to an independent nonprofit foundation. Anthropic, meanwhile, announced that Agent SDK and third-party app usage would stop drawing from Claude subscription limits, then paused that change on June 15. None of this affected me because I was already operating with my own API keys. But those shifts reinforced a durable architectural rule: the agent layer should not depend on one model provider or one commercial access mechanism.

In a few months, Hermes surpassed two hundred thousand GitHub stars and became one of the year’s most visible agent frameworks. It is model-agnostic, MIT-licensed, and supports providers ranging from OpenRouter and Nous Portal to OpenAI, Anthropic, Hugging Face, NVIDIA NIM, and custom endpoints.

The important thing is understanding what this layer does and does not do. Hermes is not the reasoning engine: it coordinates, remembers, executes skills, and maintains identity. Reasoning is delegated to the model best suited to each task. The structure is not Hermes or Claude. It is Hermes as the agent layer, with reasoning models below it, connected through APIs. My current setup is concrete: Claude Sonnet and Opus through direct API access, with accounts across several providers configured for failover. If one provider becomes unavailable, the agent is configured to switch to another and continue operating.

This separation between agent layer and reasoning engine provides long-term flexibility. The model that reasons today can change tomorrow without rebuilding the agent. Identity, memory, and skills live in Hermes; the engine is replaceable. For now, I do not run local models. That remains a future option, not a present need. Running inference inside my own perimeter will make sense when there is either sustained volume or data that must not leave it. Neither condition is validated today, so I use frontier models through APIs, with provider failover for availability.

The corpus: “ground gold”

So far, the architecture. But an agent with good architecture and no knowledge of the user is still a generic assistant. What turns it into a colleague is the corpus: curated knowledge about how the person thinks, decides, and works. This is the part I want to develop slowly, because it is where the real value lies.

In the Dominican Republic, we say something is “oro molido”, ground gold, when it is worth a great deal. The phrase fits because it separates raw material from the value created when that material is refined and made usable. Those of us who have spent time conversing with these models already have the raw material: a chat history full of evidence about how we correct and what we demand. It only needs to be processed.

The most valuable material in that history is not the documents or final outputs. It is the user’s judgment at the exact moment they correct the model. When someone tells an LLM, “No, that is not what I asked,” or “Where did you get that data?” or “This is a shortcut, and I do not accept it,” they leave a legible trace of their operational reasoning. That trace is more revealing than any polished result because it captures judgment in action, not an already edited conclusion. We correct models hundreds of times without realizing that each correction records our criteria.

Your corrections to an AI are not conversational residue; they are evidence of how you correct. And that is what an agent needs to inherit.

The method for extracting that gold has two phases. First, review the history and identify critical turns: explicit corrections, rejected approaches, demands for evidence, and declarations of principle. Second, distill the underlying operational principle from each turn and group those principles into higher-level patterns. The result is not a record of what was discussed, but a map of the criteria by which the person corrects and decides.

An isolated correction is not enough to become a permanent rule. The principle should recur across contexts, be documented with its exceptions, and survive review before it becomes part of the agent.

Applied to a real and sufficiently rich history, the method produces something uncommon: an operational profile of your correction criteria, derived from behavioral evidence rather than introspection. It is not a portrait of how we think: it is a map of how we correct, and that is the part of our thinking that matters most to an agent working for us. The corrections we make to an AI under real work pressure reveal criteria that most people never articulate about themselves. They expose the standard of evidence we demand, the shortcuts we reject, the way we reframe problems, and the point at which we reject a complacent answer. Once extracted and grouped, those patterns feed the agent’s operating profile: its system prompt, memory rules, and skills. The corrections made to the model become instructions that help keep the agent from repeating the same mistakes.

It is a cycle that closes on itself with elegance. We teach the system without intending to, through months or years of corrections. The agent learns from those accumulated corrections. And along the way, we obtains an explicit document of our operational criteria, useful beyond the agent itself. The history that seemed like conversational residue turns out to be the foundation.

That corpus also demands care. Before processing it, I exclude secrets, third-party data, and any material that should not leave the chosen perimeter.

Update note (23 Jul 2026): I put this article’s central thesis to the test against the academic literature. The research confirms that the feedback we leave behind in our chats is real, abundant, and extractable: a study of over one million conversations found explicit feedback in roughly 30% of chats, and the PRELUDE framework (Microsoft Research and Cornell, NeurIPS 2024) distills user preferences from their edits, much the same method I describe here. What no one has validated, yet, is whether that material predicts how a person reasons outside their chats. So I am correcting the claim: your history does not reveal how you think; it reveals how you correct. And for building a personal agent, that is exactly what you need. How this correction came about, and what I found along the way, is coming in the next two articles in this series: the complete method for extracting that gold and refining it, and an MIT paper I stumbled onto while digging, about something called knowledge collapse.

The decision not to buy yet

For months, my plan assumed I would buy hardware to run local models, an investment of several thousand pounds plus electricity and maintenance. After the pause, I recalculated with a more honest question: what do I need today, not what would be elegant to own? Direct access to frontier models through APIs, with provider failover, covers my real operating needs without committing capital. I have already documented what happens when API usage is not controlled, but that is an operational problem I can measure and correct.

Owned hardware becomes justified when a concrete need for sovereignty over sensitive data appears. Serverless inference with open models becomes justified when sustained volume changes the arithmetic. Neither condition is validated yet.

Buying before validating would be exactly the kind of shortcut I reject in technical decisions: a costly commitment based on assumptions without operational evidence. The architecture is designed to accommodate those engines when they are needed, without rebuilding anything. Designing for the future does not require paying for it ahead of time.

What this architecture makes possible

The value of all this is not technical; it is about ownership. The underlying idea is that each person can have their own agent, built on a foundation they have already created without realizing it through day-to-day work with trusted models. Chat histories and accumulated projects are not residue: they are the foundation.

The architecture has three virtues worth naming. The first is sovereignty over what has accumulated: what we build through conversation forms part of our accumulated intellectual work and should remain under our control. It becomes the basis of something we own, not just a feature inside someone else’s product. The second is continuity without dependence: we can keep using the same trusted models, the ones we already know and prefer, but now as components inside our own architecture. The platform no longer contains the agent, its memory, and its identity; our architecture does. We can move to whichever model is appropriate without being tied to any one of them. The third is that this is not an architecture only for specialists. Anyone who has used these chats seriously for long enough already has the raw material. The personal agent stops being a technical privilege and becomes something buildable from what we have already generated.

Returning to what is essential after the pause also meant seeing this clearly: the most valuable part of the project was not waiting to be built. It was already there. It only needed to be recognized, refined, and put to work.

By: Cesar Rosa Polanco - Written from a real experience, with artificial intelligence used as an editorial support tool.

First time here?

Explore the key topics and articles on this blog.

Start Here →