Skip to main content

Getting Started

Prerequisites​

  • Python 3.11+
  • MLflow Tracking Server (local or remote)
  • An OpenAI-compatible LLM API key (Gemini, OpenAI, Azure, Ollama, etc.)
  • For production: Kubernetes cluster with OpenShell Sandbox

Two separate services need API keys:

ServiceWhat it needsWhy
Agent Lens (agent harness)Any OpenAI-compatible API keyPowers the conversational agent that talks to you
MLflow MCP ServerOPENAI_API_KEY + MLFLOW_TRACKING_INSECURE_TLSRequired for LLM-judge scorers (evaluate_traces, register_llm_judge_scorer) and TLS connectivity to MLflow

Local Development​

Option A: Hermes (reference harness)​

git clone https://github.com/rrbanda/agent-lens.git && cd agent-lens
make dev-setup && make mlflow-start && make seed-data
make test-integration

Option B: Google ADK​

git clone https://github.com/rrbanda/agent-lens.git && cd agent-lens
cd agent-lens && cp adk/.env.example adk/.env # edit with your config

# Install dependencies (uv or pip)
uv pip install "adk/.[tracing]" # or: pip install -r adk/pyproject.toml

# Start MLflow MCP server (separate terminal)
make mlflow-start
python -m mlflow.mcp.run --tracking-uri http://localhost:5555

# Start the ADK agent (OpenAI-compatible API)
PYTHONPATH=. uvicorn adk.main:app --host 0.0.0.0 --port 8000

The ADK agent exposes:

  • POST /chat/completions — OpenAI-compatible endpoint (streaming + non-streaming)
  • GET /health — readiness probe

Uses LiteLlm to route inference through any OpenAI-compatible endpoint (vLLM, OGX, Ollama, Gemini, Azure). Skills load via SkillToolset with progressive disclosure.

Deploy to Kubernetes​

The examples below use OpenShift CLI (oc). For other Kubernetes distributions, substitute kubectl for oc.

Step 1: Create the Agent Lens auth secret​

DASH_PW=YourPassword API_KEY=YourKey LLM_API_KEY=YourGeminiOrOpenAIKey make secret-openshell

The LLM_API_KEY is mounted as OPENAI_API_KEY inside the Agent Lens pod. Any OpenAI-compatible key works (Gemini, OpenAI, Azure, Ollama).

Step 2: Configure the MLflow MCP Server​

The MLflow MCP server is a separate deployment from Agent Lens. It needs its own configuration:

# Required: Allow MCP server to connect to MLflow over self-signed TLS
oc set env deployment/mlflow-mcp MLFLOW_TRACKING_INSECURE_TLS=true -n <mlflow-namespace>

# Required for LLM-judge skills (evaluate-agent, create-judge, red-team, eval-loop):
oc set env deployment/mlflow-mcp OPENAI_API_KEY=<your-openai-key> -n <mlflow-namespace>

:::danger Critical: MLFLOW_TRACKING_INSECURE_TLS If your MLflow Tracking Server uses self-signed TLS certificates (standard on OpenShift), you must set MLFLOW_TRACKING_INSECURE_TLS=true on the MLflow MCP deployment. Without this, all MCP tool calls will hang silently — no error, no timeout message, just infinite waiting. This was the #1 deployment issue found during testing. :::

:::warning LLM Judge Model Compatibility MLflow's built-in scorers (used by evaluate_traces) default to OpenAI model names like gpt-4o-mini. If you set OPENAI_BASE_URL to a Gemini endpoint, the scorer will fail because the model name doesn't exist on Gemini's API. For LLM-judge skills, use either:

  • A real OpenAI API key (recommended), or
  • An OpenAI-compatible proxy that translates model names :::

Step 3: Build and deploy Agent Lens​

Choose your harness:

Hermes (reference):

make deploy-all && make status

Google ADK:

# Build the ADK image
podman build -f agent-lens/Containerfile.adk -t agent-lens-adk agent-lens/

# Push to your registry, then deploy
oc apply -k agent-lens/deploy/adk/

Step 4: Access the agent​

Hermes: Access the dashboard at the route URL, login with admin and your dashboard password.

oc get route agent-lens -n openshell -o jsonpath='https://{.spec.host}'

ADK: Access the ADK API server (FastAPI). For production, add an OAuth proxy sidecar for auth.

oc get route agent-lens-adk -n openshell -o jsonpath='https://{.spec.host}'

Step 5: Verify MCP connectivity​

Try: "Show me all experiments"

If it responds with a list of MLflow experiments, everything is working. If it hangs or times out, check the Troubleshooting section below.

Your First Interaction​

Once deployed, open the dashboard route and try:

"List all MLflow experiments"
"Show me the last 10 traces for my-agent"
"Evaluate my-agent using the RAG profile"
"Give me a quality dashboard across all agents"

How It Works​

Agent Lens follows a five-phase loop:

PhaseWhat HappensMCP Tools Used
ObserveDiscover experiments, tracessearch_experiments, search_traces, get_trace
EvaluateScore traces with GenAI judgesevaluate_traces, list_scorers
AnnotateLog feedback / expectationslog_trace_feedback, log_trace_expectation
QualifyPASS/FAIL against thresholdsEvaluation run metrics + trace tags
Follow upTag failures for regressionset_trace_tag, log_trace_expectation

MCP Tools Used​

All tool calls go through the official MLflow MCP server (mlflow mcp run):

ToolPurpose
search_experimentsDiscover agent experiments
get_experimentGet experiment details
search_tracesFind traces with filters
get_traceFull trace with spans
log_trace_feedbackHuman quality assessments
log_trace_expectationExpected outputs for regression
set_trace_tagTag traces (regression, reviewed)
evaluate_tracesRun LLM judges on traces
list_runsFind evaluation runs
describe_runGet run metrics and tags
list_scorersAvailable built-in judges (pass builtin: "true")
register_llm_judge_scorerCreate custom LLM judge scorers
create_runRecord evaluation runs
create_experimentCreate new experiments
delete_trace_tagRemove trace tags
get_trace_assessmentRead trace assessments
update_trace_assessmentUpdate trace assessments
delete_trace_assessmentRemove assessments
link_traces_to_runAssociate traces with runs
delete_tracesRemove traces

Troubleshooting​

MCP tool calls hang (no response, no error)​

Symptom: You ask "Show me all experiments" and nothing happens — the agent just spins forever.

Cause: The MLflow MCP server cannot connect to the MLflow Tracking Server because of TLS certificate verification failure.

Fix:

oc set env deployment/mlflow-mcp MLFLOW_TRACKING_INSECURE_TLS=true -n <mlflow-namespace>

Then restart the Agent Lens pod so it picks up the fresh MCP connection:

oc delete pod agent-lens -n openshell

LLM-judge evaluations fail with "OPENAI_API_KEY not set"​

Symptom: Skills that use evaluate_traces or register_llm_judge_scorer return errors like OPENAI_API_KEY environment variable must be set to use the openai provider.

Cause: The MLflow MCP server (not Agent Lens) needs an OpenAI API key to run LLM-based scorers. This is separate from the Hermes LLM key.

Fix:

oc set env deployment/mlflow-mcp OPENAI_API_KEY=<key> -n <mlflow-namespace>

LLM-judge evaluations fail with ChatCompletionError​

Symptom: evaluate_traces returns ChatCompletionError or Failed to invoke judge model.

Cause: MLflow's built-in scorers default to OpenAI model names (e.g., gpt-4o-mini). If you set OPENAI_BASE_URL to a non-OpenAI endpoint (like Gemini), the model name is invalid.

Fix: Use a real OpenAI API key, or set up an OpenAI-compatible proxy that handles model name translation.

Affected skills: evaluate-agent, create-judge, red-team, eval-loop. All other skills (12 out of 16) work with any LLM provider.

Pod keeps getting evicted (ephemeral-storage)​

Symptom: Agent Lens pod enters Error or Evicted state with The node was low on resource: ephemeral-storage.

Cause: The Kubernetes node has too many container images or orphaned pods consuming disk space.

Fix:

# Delete failed pods that are consuming metadata space
oc delete pods --field-selector=status.phase=Failed -n <namespace> --wait=false

# If the pod keeps landing on the same bad node (PVC-bound), delete the workspace PVC
# to let OpenShift schedule on a healthy node:
oc delete pod agent-lens -n openshell
oc delete pvc workspace-agent-lens -n openshell
# The sandbox controller will create a new PVC on a healthy node

Dashboard returns 302 but won't load​

Symptom: The route is accessible (302 redirect) but the dashboard doesn't load.

Fix: The dashboard is on port 9119. Verify the route targets this port:

oc get route agent-lens -n openshell -o jsonpath='{.spec.port.targetPort}'

"Show me all experiments" works but other skills time out​

Symptom: Simple MCP queries work but complex skills (executive-summary, compliance-export) time out in CLI mode.

Cause: Some skills require multiple sequential MCP calls. CLI mode has shorter timeouts than the dashboard.

Fix: Use the web dashboard instead of CLI mode for complex skills. The dashboard handles long-running MCP workflows better.

Which skills work without an OpenAI key?​

These 12 skills work with any LLM provider (Gemini, Ollama, etc.) because they only use MCP data tools, not LLM judges:

SkillWhat it does
trace-explorerList experiments and traces
quality-dashboardFleet health overview
analyze-sessionTrace analysis
review-traceDeep trace inspection
create-regressionFlag traces as regressions
compare-evaluationsSide-by-side run comparison
cost-qualityQuality vs cost analysis
audit-trailQualification decision history
agent-registryFleet inventory
executive-summaryBoard-ready health summary
compliance-exportJSONL/CSV export
aggregate-tracesError rates, latency trends

These 4 skills require an OpenAI-compatible API key on the MLflow MCP server for LLM judges:

SkillWhat it does
evaluate-agentRun scorers against traces
create-judgeRegister custom LLM judge scorers
red-teamAdversarial safety evaluation
eval-loopFull EDD improvement cycle