Skip to main content

Architecture

System Overview​

Design Principles​

  1. MCP-native — All data access through official MLflow MCP tools. No direct SDK calls at runtime.
  2. Skills as prompts — Each skill is a SKILL.md file, not code. The LLM interprets it and calls MCP tools.
  3. Upstream-first — Never fork MLflow MCP. Missing features go upstream.
  4. Conversational-first — The UI is chat. Structured output (tables, reports) rendered in conversation.
  5. Sandbox-first — The qualification agent itself runs inside an OpenShell sandbox. Security is not a bolt-on.

Security Model: OpenShell Sandbox​

Agent Lens runs inside an OpenShell sandbox — the same defense-in-depth isolation used for production agent workloads. The qualification agent is sandboxed, not just the agents it evaluates.

Infrastructure → Sandbox → Harness → Skills → Model
K8s OpenShell Agent Agent Lens LLM
Runtime

Defense-in-depth layers​

LayerMechanismWhat it does
NamespacesPID, mount, network, userRestricts the agent's view of the system
Landlock LSMKernel-enforced filesystem ACLsEven root inside the namespace can't escape allowed paths
Seccomp-BPFSystem call filteringBlocks ptrace, mount, memfd_create, raw sockets
Capability droppingEmpty bounding set"Root" in the namespace has no real capabilities
L7 network proxyBinary identity bindinggit can reach github.com; curl from the same sandbox cannot
Resource controlscgroups v2CPU, memory, and I/O limits prevent runaway processes

Sandbox vs. harness — separate concerns​

The sandbox is subtractive — it constrains what the agent can do and limits blast radius. The skills are additive — they layer on knowledge and MCP tool access to increase competence. These have different failure modes:

  • Sandbox failure = the agent did something it shouldn't have been able to do
  • Skill failure = the agent did something poorly that it should have done well

The sandbox also serves as a recorder — a neutral observer that can attest to what the agent did, providing provenance information independent of the agent runtime's self-reporting.

Kubernetes integration​

On Kubernetes, Agent Lens deploys via the agent-sandbox-operator which manages the sandbox pod lifecycle (warm pools, PVCs, rescheduling). The OpenShell supervisor inside the pod enforces the actual security boundary — filesystem policy, egress enforcement, credential isolation, and OCSF audit events.

Component Details​

Agent Harness (runtime) — pick yours​

Agent Lens requires an MCP-capable agent harness — any runtime that can:

  • Call MCP tools (streamable HTTP or stdio transport)
  • Load and interpret SKILL.md prompt documents
  • Maintain session context across multi-step workflows
  • Provide a chat interface (web dashboard or CLI)

Agent Lens ships with two validated harnesses — the skills, soul, and config are portable artifacts that work with either runtime (and any other MCP-capable agent).

HarnessContainerStartupPortMCP wiringSkill loading
HermesContainerfilestartup.sh9119config.yaml mcp_serversBuilt-in skill curator
Google ADKContainerfile.adkstartup-adk.sh8000McpToolset in agent.pySkillToolset (agentskills.io) + LiteLlm model connector
Harness-independent (keep)Harness-specific (swap)
Filesskills/*.md, soul.mdContainerfile, startup.sh, agent.py
WhyStandard MCP tool patterns any client can executeRuntime-specific packaging and lifecycle

The ADK harness uses a custom FastAPI server (main.py) with an OpenAI-compatible /chat/completions endpoint — following the agentic-starter-kits BYOC pattern. It uses LiteLlm to route inference through any OpenAI-compatible endpoint (vLLM, OGX, Ollama, Gemini, Azure), McpToolset for MLflow MCP tools, and SkillToolset for the 16 SKILL.md files. MLflow tracing is auto-enabled via mlflow.litellm.autolog() when MLFLOW_TRACKING_URI is set.

Compatible runtimes also include Claude Code, OpenClaw, Goose, or any custom MCP-capable agent. Choose a commodity runtime — the qualification logic lives in the skills, not the harness.

Agents being evaluated (target agents)​

Agent Lens evaluates any agent that sends traces to MLflow, regardless of framework:

FrameworkMLflow integration
LangGraphmlflow.langchain.autolog()
Google ADKmlflow.litellm.autolog() — automatic CHAT_MODEL + TOOL spans
LangChainmlflow.langchain.autolog()
CrewAImlflow.crewai.autolog()
OpenAI Agents SDKmlflow.openai.autolog()
AutoGenmlflow.autogen.autolog()
LlamaIndexmlflow.llama_index.autolog()
Custom@mlflow.trace decorator or REST API

MLflow MCP Server​

Official mlflow mcp run providing 19 tools:

tools:
include:
# Observability
- search_experiments
- get_experiment
- search_traces
- get_trace
- list_runs
- describe_run
# Evaluation
- evaluate_traces
- list_scorers
# Annotation
- log_trace_feedback
- log_trace_expectation
- set_trace_tag
- delete_trace_tag
# Assessment
- get_trace_assessment
- update_trace_assessment
- delete_trace_assessment
# Scorer management
- register_llm_judge_scorer
# Run-trace association
- link_traces_to_run
# Lifecycle
- delete_traces
- create_run
- create_experiment

MLflow AI Gateway (built into MLflow)​

MLflow AI Gateway runs as part of the MLflow Tracking Server and provides:

  • Governed LLM access for all providers (OpenAI, Anthropic, Gemini, etc.)
  • Automatic tracing of all LLM requests with token counts
  • Automatic evaluation — LLM judges run as traces arrive
  • RBAC and credential management

No custom gateway service needed — Agent Lens consumes these features directly.

Deployment Topology​

Both harnesses share the same MLflow MCP server, skill ConfigMaps, and OpenShell sandbox security model. You can run them side by side or choose one — the skills are identical.

ADRs​