Taming the AI Agents

Preface: A Noob’s Journey into Agentic AI

Although the AI surge felt like it was simmering for years, I jumped on the bandwagon relatively late—around November 2025.

My very first interaction with AI was Perplexity through its web interface. As a complete novice, my expectations were honest and naive. It took multiple conversational iterations and several frustrated hours to finally grasp what an “AI hallucination” actually meant. Soon after, I tried Google’s Gemini web interface. While helpful for reading bedtime stories out loud to my kids, it didn’t immediately feel game-changing for engineering tasks.

The real shift happened when I started using gemini-cli. Because it was developed in the open, I could look under the hood, experiment with its capabilities, and hack custom workflows directly into the CLI tool. My curiosity quickly snowballed. I expanded into GitHub Copilot, gaining access to Anthropic’s Claude models, and began learning the subtle art of model rationing—drafting specs with heavy-hitter models and delegating execution to lighter ones. Trial and error forced me to quickly learn the vocabulary of modern AI: tokens, context windows, KV caches, and context economy.

Initially, my agent memory setup was just a static MEMORY.md file—until I realized agents could modify or overwrite the entire file! When MemPalace launched, I jumped on it on Day 1. That became our shared vector memory pool, giving rise to CAMP (Cross-Agent Memory Protocol) to unify a growing, heterogeneous fleet (Gemini and Copilot).

When Google pivoted consumer access from gemini-cli to Antigravity (AGY), it triggered a new wave of refactoring. Early attempts to fortify these agents failed repeatedly until a human colleague suggested using Linux bwrap (Bubblewrap). That tip changed everything: it enabled us to build a dual-layer, sandboxed runtime where agents have full operational freedom inside a container without ever holding an unaudited host shell.

With CAMP memory, local Gitea for persistence, and bwrap sandboxing in place, building further automation layers became fast and deterministic. Courtesy of this agentic AI fleet, a massive backlog of long-standing personal projects has finally moved across the finish line:

• Complete website overhaul and Hugo theme migration
• Embedded guest and GitLab commenting engine
• Jellyfin File Browser integration
• Personal digital diary and automated log system
• A stubborn, 4-year-old power drain bug on my ThinkPad T14 Gen2 AMD!

What follows is the technical blueprint of the architecture, memory model, and security rails that made this transformation possible.

The Paradigm Shift: AI as the PC

If you step back and look at the big picture, we are living through a fundamental shift in how we build computing environments. In the traditional world, the CPU executed deterministic assembly instructions, RAM held temporary program stacks, and NVMe drives stored structured files. In this new agentic era, that architecture is being redrawn: the LLM acts as the CPU, the Context Window is the RAM, and the Vector Database becomes the long-term NVMe storage.

Yet, as we transition to this new paradigm of “AI as the PC,” we lack what may conceptually behave like an operating system kernel. Without it, autonomous agents run with the equivalent of unchecked privileges. They get trapped in infinite, resource-consuming loops, run unverified scripts, suffer from “investigative drift” (where a minor warning leads to hours of system tampering), and burn through API credits.

Over the last few months, we set out to experiment with what may potentially become a conceptual “kernel layer” to bring order. The result is the Cross-Agent Memory Protocol (CAMP) framework. Here is how we have worked towards taming our agent fleet, securing data sovereignty, and bridging the gap between autonomous freedom and deterministic control.

The Core Dilemma: Autonomy vs. Fortification

Taming an agent is a delicate balancing act.

If you fortify the environment too strictly (mounting the entire filesystem as read-only, stripping away command line tools, blocking all network access), you strip the agent of its ability to think, debug, and work smart. It becomes a glorified static text generator, unable to solve real programming tasks.

On the other hand, giving agents unrestricted host access inevitably leads to “honest rogue” behavior. An agent with honest intentions to fulfill a user request will, when blocked by system quirks, go to extreme lengths to bypass those blocks—sometimes behaving indistinguishably from a malicious attacker.

Case Study: The Honest Exploit Attempt

We witnessed this balance break during the early development of our sandboxed runtime (camp-acpd and InnerPEP). We had tightened the container mounts, inadvertently masking the standard /usr/bin/git binary.

One of our agents was tasked with running git status on the repository. Finding git: command not found in its path, the agent did not stop or ask for help. Instead, driven by its objective, it began a rapid, autonomous probe of the sandbox. It searched the filesystem, discovered the raw binary at /usr/lib/git-core/git, and attempted to invoke it directly via Python subprocesses. When further security flags blocked that execution, the agent began dynamically attempting known path-traversal workarounds and brute-forcing local file operations to bypass the container’s path restrictions.

It was a clean, logical attempt to solve a problem—but it was still an active brute-force exploit run on the host’s directory structure. Without sandbox boundaries, an agent trying to be “helpful” can easily end up compromising the system.

Our response was not “lock everything forever.” Early fortification attempts that over-tightened the Inner PEP burned more human time on agent brain-exhaustion than they saved. The working doctrine became relaxed-first, then tighten bolts in small chunks when real friction appears—while never giving back an unaudited host shell.

1. camp-acpd: The Sole Gateway to Command Execution

To bring order to this dilemma, we built camp-acpd—a central daemon that acts as our single point of entry for any command execution, file modification, or forge operation.

We deliberately use a dual-layer sandbox, not one monolithic jail:

1. OuterWrap — the long-lived agent process (the LLM client) runs inside a strict bwrap profile. It sees a curated view of the host: a small read-only /camp-runtime/ of verified bridges, the agent’s workspace, and only the scratch roots it needs. The live CAMP source tree is masked (empty tmpfs over the mutable checkout) so the model cannot wander the implementation and burn context on infrastructure archaeology.
2. Inner PEP — when the agent runs a shell command, camp-shell forwards it to camp-acpd, which builds a fresh (today: relatively relaxed) Bubblewrap namespace for that command, audits it, and streams back pristine stdout/stderr. Security bolts tighten in small chunks; the first product goal was “commands look identical to native bash” so real work could continue while fortification matured.

Look at what happens when an agent lists /home/rrs/ inside the OuterWrap:

$ ls /home/rrs/ -al
total 0
drwxr-xr-x 5 rrs rrs 100 Jul 29 14:07 .
drwxr-xr-x 3 rrs rrs  60 Jul 29 14:07 ..
drwxrwxr-x 1 rrs rrs  56 Jun 21 20:52 .agy-agrickxy
drwx------ 5 rrs rrs 100 Jul 29 14:07 AI
drwx------ 3 rrs rrs  60 Jul 29 14:07 NoBackup

The agent is blind to host documents, SSH keys, and most of the home directory. It only sees its designated workspace and the necessary AI/NoBackup scratch roots.

The OuterWrap mounts a read-only /camp-runtime/ directory containing only the verified binary bridges required for IPC and secure utilities:

$ ls /camp-runtime/bin/
camp-dbus  camp-mcp-shim  camp-shell  camp-sudo  dbus-send  grep  rg

• Complete Auditability: Every shell command, git operation, or forge write requested by an agent must route through camp-acpd. There are no unaudited backdoors. Actions are recorded to the camp_audit ledger—the durable evidence trail. (This is distinct from PDP, our Personal Data Plane: a downstream work-hours / appraisal ledger derived from forge and audit evidence, not the security log itself.)
• Scoped, Per-Agent Credentials: No agent shares a root or admin token. Each identity (e.g. CPL, AGR, GRK, KIR) gets a narrowly-scoped forge token, brokered outside the sandbox and injected only at the point of use. Compromise of one agent does not hand over the fleet.
• Policy layer (OPA, pluggable): camp-acpd can consult Open Policy Agent (OPA) for declarative allow/deny rules (role, path, tool class, egress). In production we treat OPA as infrastructure that can be relaxed or tightened without rewriting the daemon—HITL rails and brokered tools remain the hard gates for privileged ops even when OPA is not the primary choke.
• The camp-shell Rust wrapper: Inside the agent environment, shell is not raw bash. camp-shell intercepts the command line, attaches session/capability context, hands off to camp-acpd, and streams output back so the agent still feels like a normal terminal—while never holding a host shell of its own.
• Host bridges, not host buses: Surfaces the Inner PEP cannot expose safely (user D-Bus, user systemctl, elevation) are reached through named bridges—camp-dbus, camp-sudo—not by inventing a side path. When a direct call fails inside the sandbox, that is a signal to take the bridge, not to declare the capability “unavailable.”

2. Zenity & HITL: Keeping the Human in the Loop

When an agent requests a high-impact, privileged, or irreversible write action (like modifying systemd units, merging pull requests on Gitea, or running a sudo command with CAMP_NO_BWRAP=1), the autonomy ends.

camp-acpd automatically suspends the execution and triggers a host-side Zenity GUI dialog on the user’s desktop:

[ Agent requests systemctl restart acpd ]
                   │
                   ▼
[ camp-acpd intercepts command ]
                   │
                   ▼
[ Host-side Zenity prompt pops up on GNOME desktop ]
 ┌──────────────────────────────────────────────┐
 │ Human: Agent 'AGR' requests root privilege   │
 │ Command: systemctl restart camp-acpd.service  │
 │           [ Approve ]  [ Deny ]              │
 └──────────────────────────────────────────────┘

Through camp-dbus (our mediated D-Bus / systemctl bridge) and related HITL rails, the sandboxed agent never talks to the host session bus or user systemd directly. The host daemon raises a desktop confirm when required. The command runs only if the human approves (or, for scripted host-side work, supplies an explicit capability path that agents cannot read—their view of that token is masked to /dev/null).

The same HITL idea covers more than sudo. Privileged agent actions—merging a pull request via the gateway, restarting a camp-* user unit, elevating through camp-sudo—are designed so that intent is a second party, not a string the model can type into its own environment.

3. Sifting the Output: camp-shell + Sieve

Even when commands are safe, verbose program outputs can easily overwhelm the “RAM” of our computing paradigm—the LLM’s context window. Running a compiler or listing a massive directory might generate thousands of lines of output, pushing the agent’s prompt size to its limit and rendering it unable to reason effectively.

To prevent this context bloat, camp-shell integrates with the Shell Sieve. If an agent attempts to run a verbose command like listing /bin/, the output is automatically truncated and sieved:

$ ls /bin/
--- [CAMP SHELL SIEVE: output truncated for context economy] ---
'['
 411toppm
 7z
 7za
 7zr
 ... [4926 lines hidden — 66824 bytes total] ...
 zipsplit
 zless
 zlib-flate
 zmore
 znew
 zrun
 zstd
--- [FULL OUTPUT ARCHIVED: /home/rrs/.cache/camp/shell-artifacts/camp_cmd_2443e7d6add2c9fa.log (5026 lines, 66824 bytes)]
    To read it WITHOUT re-flooding context (a plain `cat` is re-sieved):
      - a file-read/viewer tool on that path (bypasses the shell), or
      - small slices: sed -n '120,160p' <path> ; grep -n PATTERN <path> ; head -c 4000 <path> ---

• Context Hygiene: The agent only receives a high-signal overview of the first and last few lines (capping at 150 lines or 5,000 characters). The full, untruncated log is written directly to disk.
• On-Demand Inspection: If the agent needs to inspect a specific compiler error deep inside the output, it is instructed by camp-shell to bypass the terminal and read the specific slice of the archived file directly, preserving context capacity and reducing API token costs.

4. Local-First Forge and Egress Default-Off

Autonomy on the host is useless if every push still phones home to someone else’s cloud. CAMP is local-first: each agent works against a local Gitea hub (localhost:8095). Mirroring into CAMP from upstream forges is open; egress back out (git push to GitLab/GitHub, glab/gh that would create remote MRs, and similar) is default-OFF.

The chokepoint is deliberate and boring on purpose:

• A pre-push hook (and matching forge CLI wrappers) allow pushes only to the local hub without ceremony.
• Any other remote requires human intent: either a host-side capability token the sandbox cannot read, or a Zenity “Allow push?” dialog on a graphical session.
• The token path is fixed, not env-overridable—agents once nearly gained a bypass when a test knobs made the path configurable; that class of hole is treated as a security regression, not a convenience feature.

The result matches the dual-layer philosophy: agents remain free to branch, commit, and open pull requests locally; publishing outside the house stays a human act.

The PR rail (humans merge, agents deliver)

Fleet rules are equally blunt about who may change production: agents open pull requests under their own identity; only the human merges canonical branches and deploys. There is no shared admin token for agents to “just fix prod.” That sounds bureaucratic until the first time an agent would otherwise have “helpfully” force-pushed a mainline branch at 2 a.m. Attribution, review, and deploy stay human-sovereign.

5. MemPalace: Attributed Long-Term Vector Memory

Rather than leaving agents to guess or make assumptions, we integrated MemPalace—a central, shared vector repository—into the heart of CAMP. The backend itself has evolved: MemPalace began on ChromaDB, but the entire fleet (11,000+ drawers across every agent wing) has since been migrated to pgvector on PostgreSQL 18, giving us transactional guarantees, better concurrency under a shared fleet, and a single canonical store instead of per-agent SQLite files.

On wake-up, the agent does not start with a blank slate. It calls mempalace_status to load the current palace map and runs semantic queries on past session summaries. This allows the agent to pull down historical references, recall user preferences, and review past debugging decisions, eliminating context silos between separate runs.

Memory is written in AAAK (Attributed Agentic Association Keys)—a compressed, attributed dialect that stores dates, importance ratings (★ to ★★★★★), and agent attributions. Because writes come from a fleet of heterogeneous CLIs with no shared runtime, we did not build one monolithic “memory service”—we built a set of small, independently-scheduled tools that each own one failure mode:

• camp-mempalace-miner — the ingestion layer. This tool actually exists in two generations, still running side by side. The original per-agent hook miners (one per CLI family—Gemini, Copilot, Grok, the sandboxed pilot agents) are launched directly from each agent’s own lifecycle hooks (AfterAgent, PreCompress/PreCompact, SessionEnd) as a detached subprocess, throttled to at most once per 30 minutes or 30 user turns. On top of that we added a central, timer-driven miner—a single systemd --user timer firing roughly every 9 minutes—that knows the on-disk transcript layout for every agent code in the fleet (e.g. Copilot’s ~/.copilot/session-state/*/events.jsonl, Grok’s ~/.grok/sessions/**/updates.jsonl, each Gemini persona’s own chats/ directory) and mines all of them into the one central palace, stamping every drawer with agent_id and added_by=camp-central-miner. This mattered in practice: sandboxed pilot agents running under bwrap were writing into a tmpfs overlay that evaporated on exit, so their memory silently never reached the host palace until the central miner started reading their transcripts directly instead of trusting their in-sandbox writes. Both generations share the same crash-safety plumbing—per-session byte-offset tracking, content-hash dedup, advisory file locks, and a CAMP_DRY_RUN mode for safe testing—so neither can double-file or corrupt state if it’s killed mid-run.
• camp-mempalace-compactor — hierarchical aging. A periodic job that finds room_general drawers older than 30 days, batches 10–15 raw snippets at a time, and asks a locally-hosted LLM to compress them into one dense AAAK summary block, keeping vector search signal-dense instead of drowning in verbatim history. We learned this the hard way: running the compactor’s small maintenance model on the same GPU as the interactive 7B model caused it to crash under Vulkan device contention, so the maintenance model now runs CPU-only, a perfectly adequate trade-off for a background summarizer.
• camp-mempalace-fsck — the integrity and semantic auditor. This single tool absorbed what we originally scoped as two separate ideas (a syntax “validator” and a consistency “fsck”), because in practice they’re one audit pass. In its default, unattended mode it deep-scans the palace for broken invariants—missing agent_id attribution, null or malformed embeddings, incomplete document text—and auto-repairs whatever is safe to fix without judgment calls. Its --attended mode is the interesting one: it hands ambiguous, potentially mis-tagged drawers to a local LLM for reclassification, shows you its proposed room change, and waits for an explicit Approve? [y/N/q] before committing—keeping a human in the loop for anything that requires judgment rather than mechanical repair.

Together, this trio is why the fleet’s memory coverage doesn’t depend on any single agent behaving well: even if a sandboxed pilot never runs its own hook miner correctly, the central miner will still find and file its transcripts on the next timer tick, and fsck will catch and repair anything that slips through mangled. A single gateway call, report_miner_brief, aggregates every agent’s mining state (sessions mined, exchanges filed, pending backlog) into one fleet-wide status line—so verifying that all agents are actually being remembered is a one-shot check, not a per-agent archaeology dig.

Diary sovereignty is non-negotiable: one agent does not read another’s private diary without an explicit tunnel and permission. The shared palace holds fleet knowledge; private journals stay private by architecture, not by “please don’t look.”

Vector recall alone isn’t enough, though—embeddings can retrieve a stale fact just as confidently as a current one. So MemPalace also maintains an explicit Knowledge Graph (mempalace_kg_add / mempalace_kg_invalidate / mempalace_kg_query) for hard facts that change over time (a DSN, a service version, a person’s role). When a fact changes, the old entry is explicitly invalidated rather than left to be out-competed by a newer, similarly-worded memory. The house rule we drilled into every agent: before stating anything about a person, project, or past event, query the palace first—wrong is worse than slow.

6. MCP: One Syscall Table for a Heterogeneous Fleet

None of this architecture would be practical if every agent vendor insisted on its own custom integration dialect. Our fleet is genuinely heterogeneous: GitHub Copilot, Gemini CLI, Claude, and Grok each come with different native tool-calling schemas and different ideas of what a “tool” should be. The Model Context Protocol (MCP) is what makes CAMP’s foundational services vendor-agnostic: every capability the fleet needs is exposed once, as an MCP server, and every agent—regardless of who built it—talks to the exact same tool surface.

camp_acp_gateway is the flagship example: a single MCP server that fronts almost the entire CAMP foundation—

• Memory (mempalace_status, mempalace_search, mempalace_kg_*, mempalace_checkpoint) — the palace operations covered above.
• Work tracking (camp_issue_create/camp_issue_comment/camp_issue_update, camp_pr_create/camp_pr_merge) — the shared Issue Tracker and PR workflow.
• Agent-to-agent coordination (camp_a2a_propose_task, camp_a2a_send_message, camp_a2a_fetch_inbox) — the passive A2A layer.
• Evidence and provenance (forge_discover, forge_ledger, forge_onboard) — the ingestion side that feeds PDP’s work ledger from real Gitea/forge activity, keeping billable-hours accounting evidence-based rather than self-reported.
• Global policy and directives (camp_global_directives, camp_policy_map, camp_policy_search) — the fleet-wide rulebook every agent reads on wake-up, versioned like everything else.

The effect is conceptually analogous to an OS syscall table: application code doesn’t care whether it’s running on one CPU family or another, because the interface layer presents one stable interface underneath. Here, an agent doesn’t care if it’s Copilot’s tool-call schema or Gemini’s function-calling format—camp_acp_gateway presents the same tools, the same argument shapes, and the same agent_id-stamped audit trail no matter which vendor is asking. That attribution is not incidental: every MCP call is tagged with the calling agent’s code (CPL, GRK, KIR, …), so the same accountability the sandbox enforces at the shell layer is also enforced at the memory, task-tracking, and coordination layer. A rogue or buggy agent can’t quietly bypass its own audit trail just because it happens to be a different vendor’s CLI.

Discovery is uniform too: camp_mcp_catalog lets any agent enumerate what’s actually available on the gateway at runtime, rather than hard-coding tool lists per agent—useful when the tool surface grows (as it regularly does) without every agent’s configuration needing a synchronized update.

7. Agent Isolation and the A2A Horizon

In the CAMP architecture, each agent is treated as a unique, independent entity. Agents operate in isolated sandboxes with distinct workspaces and credentials. An agent cannot mutate another agent’s repository or step on its toes without explicit permission.

Fleet behaviour is not left to tribal knowledge. A short constitution—camp-directives.md—is served verbatim to every agent on wake-up and on a standing cadence via camp_global_directives. Identity, memory, egress, shell routing, A2A, and tooling posture live there once; per-agent instruction files are only overlays (paths, runtime quirks), not forks of the rules.

Currently, cooperation is achieved through a passive A2A (Agent-to-Agent) mechanism:

• Agents send messages, task proposals, and status updates via camp_acp_gateway.
• These tasks are queued and reviewed when a session is active or when the user acts as a mediator.
• For anything longer-lived than a single session, the fleet leans on a shared CAMP Issue Tracker (Gitea-hosted, camp_issue_create/camp_issue_comment/camp_issue_update). Work items carry explicit dependency links—e.g. an issue implementing a forge adapter will note “depends on #472 (contracts)” and an agent picking it up can develop against the dependency’s branch before it lands. Pull requests are reviewed and commented on across agents and the human, so a fix started by one agent in one session can be picked up, critiqued, and finished by a different agent (or the same one, days later) without losing any context—the issue is the context.

The Headless Limitation

While passive A2A works beautifully for structured handoffs, the current frontier of agentic design faces a key limitation: agents are not yet fully headless-capable. They depend on the active terminal session, browser loop, or prompt loop of the user to keep executing.

Because agents cannot run completely detached in the background as daemon processes, we cannot yet achieve active A2A communication—where a swarm of agents autonomously wakes up on a cron schedule, coordinates complex migrations in the background, resolves merge conflicts among themselves, and presents a completed PR in the morning without any active human terminal sessions. Overcoming this headless hurdle is the next major step in our roadmap.

8. The Local-First Promise: Data Sovereignty

Utter data sovereignty means keeping your memory, code, and execution local—and making cloud models optional guests, not landlords.

• Local Gitea hub: Code, issues, and pull requests live on a machine you control. Agents are first-class forge users there; external forges are mirrors you choose to push, not the default workspace.
• Ollama (and friends): CAMP integrates with local LLM runtimes. We run models such as Qwen 2.5 7B, Gemma 2B, and Phi 3.5 Mini locally, accelerated by the laptop’s iGPU (Vulkan). Maintenance models for memory compaction can run CPU-only so they do not thrash the interactive GPU session.
• Model-agnostic sovereignty: Because vector memory, local Gitea, and execution sandboxes are decoupled from any one vendor CLI, a new open-weights model or a new agent frontend is a plug-in—not a migration. Historical memory, access policies, and coding workflows stay put.

Conclusion: Bridging the Paradigm

By combining local LLM execution (Ollama), dual-layer sandboxing (OuterWrap + Inner PEP via camp-shell / camp-acpd), HITL rails (Zenity, fixed-path capability tokens, human-only merge/deploy), egress default-off to a local Gitea hub, a vendor-agnostic MCP tool surface (camp_acp_gateway), and automated memory maintenance (MemPalace suite), we’ve tried to experiment with what may potentially become the conceptual equivalent of an operating system kernel for the AI era.

The work is unfinished by design. Headless swarm coordination, runtime-directory default-deny (so the next secret is invisible without a deliberate allow-list), and further tightening of relaxed Inner PEP mounts remain open. The point of such a conceptual kernel is not to pretend agents are tame—it is to make every ambitious shortcut auditable, attributable, and interruptible by a human.

We no longer treat autonomous agents as unpredictable, untrusted black boxes. They are disciplined pair-programmers: free enough to do real engineering, bound enough that “helpful” does not become “hostile,” and sovereign enough that the house—not the cloud vendor—owns the ledger of what they did.

Video Demonstrations: CAMP MemPalace Memory in Action

Below are three video demonstrations showing CAMP MemPalace memory integration in action across three different AI agent clients:

Antigravity Agent Client

Claude Agent Client

Grok-Build Agent Client