Your SOC Agent's Final Answer Is Lying to You

TL;DR — A TF-IDF + logistic-regression baseline scores a perfect 1.00 macro-F1 on alert severity classification in our benchmark. Does that make it a competent security analyst? Obviously not. Final-answer metrics saturate quickly and tell you almost nothing about whether an agent can run a real investigation. We built SOC-AgentBench, an episode-based benchmark that grades the whole trajectory of a SOC investigation — evidence, timeline, tool discipline, and safety — and found that a fine-tuned 1.5B agent that succeeds at 88% of its tool calls still identifies the correct root cause only 58% of the time. The 30-point gap between tool syntax and actual understanding is exactly what final-answer benchmarks hide.

1. A perfect score that means nothing

While building baselines for our SOC benchmark, we hit a number that should make anyone suspicious: a TF-IDF + logistic-regression classifier reached 1.00 macro-F1 on alert severity classification on the test split (eval_results.json).

A model with no understanding of attacker behavior, no ability to read a log, and no concept of what LSASS is, achieved a perfect score on the metric most SOC ML papers still report.

There are two ways to read this:

1. "Great, severity classification is solved!"
2. "The metric is measuring the wrong thing."

The second reading is the correct one. Severity labels correlate strongly with surface vocabulary of the alert text ("mimikatz", "lsass", "psexec"), so a bag-of-words model separates them cleanly — especially on a small test split. But a SOC analyst's job was never to map alert text to a severity string. It is to investigate: pull the right logs, pivot on the right host, build a timeline, distinguish true positives from noise, and commit to a root cause — without inventing evidence and without nuking a production server "just to be safe".

None of that appears in a final-answer score. This is the security-domain version of a shift the broader agent community made this year: Hugging Face's Is it agentic enough? benchmark argues that for capable models, task completion saturates near 100% and stops being informative — what matters is the work: turns, tokens, errors, and the path taken. We think security is the domain where this argument matters most, because in a SOC the path is the product. An analyst who reaches the right verdict by guessing is a liability, not an asset.

2. Why security agents specifically need trajectory evaluation

Three properties make SOC work uniquely hostile to final-answer evaluation:

The verdict is low-entropy; the investigation is not. Most alerts resolve to a handful of incident types. An agent can reach 70%+ verdict accuracy through priors alone ("PowerShell alert → probably execution technique"). Only trajectory metrics expose whether it actually looked at the evidence.

Hallucinated evidence is worse than no evidence. An LLM that fabricates a log line into its incident report doesn't just fail — it actively poisons the downstream response process. A human analyst will act on that fabricated artifact. Any SOC agent evaluation that doesn't measure hallucinated evidence rate is ignoring the primary failure mode of the technology.

Actions have blast radius. Security agents can isolate hosts, disable accounts, and block IPs. Recent work on agent tool-layer security shows how easily agent behavior is manipulated: the MCPTox benchmark reports tool-poisoning attack success rates above 60% across major LLM agents, with the most capable models often more compliant with malicious tool metadata, not less (threat-modeling analysis, ACL 2026 industry track). If your evaluation has no concept of a forbidden action, you have no way to detect the agent that reaches the right answer while doing something catastrophic along the way.

3. Anatomy of an evaluable investigation

SOC-AgentBench packages each investigation as a self-contained episode: 120 episodes across 12 attack scenarios (credential dumping, lateral movement, C2 beaconing, cloud persistence, …), with logs from 9 sources (Sysmon, Zeek, Suricata, Wazuh, Windows, Linux, K8s Audit, CloudTrail, Azure) and ground-truth labels for 15+ MITRE ATT&CK techniques.

The key design decision is that every episode carries both the world and the answer key:

{
  "episode_id": "soc-000123",
  "scenario_name": "lsass_memory_dump",
  "initial_alert": { "title": "...", "severity": "high" },
  "available_tools": ["search_logs", "get_host_context", "..."],
  "log_corpus": [...],
  "asset_inventory": {...},
  "hidden_ground_truth": {
    "incident_type": "credential_access",
    "attack_techniques": ["T1003.001"],
    "compromised_hosts": ["WS-104"],
    "root_cause": "...",
    "is_true_positive": true
  },
  "gold_evidence": [...],
  "gold_timeline": [...],
  "forbidden_actions": ["isolate_host_without_evidence", "..."]
}

The agent gets 7 tools (search_logs, get_host_context, get_user_context, lookup_mitre_technique, enrich_ip, build_timeline, submit_incident_report) that execute against episode-local data only. No network, no live SIEM, no flakiness: every tool call is deterministic and auditable, which means every run is reproducible and every trajectory can be scored offline.

Because the episode contains gold_evidence, gold_timeline, and forbidden_actions — not just a final label — we can grade the process:

The last two rows are the ones we've never seen in a classical security-ML paper, and they are the ones that decide whether you'd let the agent anywhere near production.

4. What the numbers say

We evaluated two very different systems on the test split: the classical baseline stack (soc-analyst-baseline: BM25 retrieval + rule-based ATT&CK mapper + TF-IDF severity classifier) and a QLoRA fine-tune of Qwen2.5-1.5B-Instruct for structured tool calling (soc-analyst-tool-use, r=16, α=32, trained on the 84-episode train split).

Two final-answer metrics (yellow) look like a solved problem. The trajectory metrics (blue) tell a different story. And the interesting structure appears when you order the agent's metrics by investigation depth:

This is a monotonic decay, and we believe it is the single most useful signal a trajectory benchmark produces. Read bottom-up, it says: the model has learned the syntax of investigation but not its semantics. It reliably produces well-formed search_logs calls (0.88), usually pulls relevant events (0.72), often labels the technique correctly (0.68) — and then, when it has to compose all of that into a causal explanation, it succeeds barely better than a coin flip on a multi-way problem (0.58).

A final-answer benchmark would have reported one number from this table — probably the F1 — and hidden the decay entirely. Worse, a verdict-only benchmark ("is this a true positive?") would likely have scored this same agent in the 80s and called it deployment-ready.

Honest caveats. The test split is 18 episodes — small enough that every number above carries wide error bars, and the baseline's perfect 1.00 should be read as "saturated on this split", not "solved". That is itself part of the argument: small, single-number security benchmarks invite overclaiming, which is why per-sample trajectory scoring and public episode data matter more than the headline metric. All artifacts, including the raw eval_results.json, are on the Hub for anyone to re-run and dispute.

5. The safety dimension: forbidden actions and poisoned tools

Every SOC-AgentBench episode declares forbidden_actions — responses that would be operationally harmful given the evidence available, like isolating a host on a probable false positive. The unsafe action rate turns "the agent did something scary" from an anecdote into a metric you can regress against.

This connects directly to the tool-poisoning results mentioned above. The defense frameworks emerging this year (metadata pinning, allowlisting, runtime tool-layer inspection) all need a measurement layer to prove they work; a trajectory benchmark with explicit forbidden actions provides exactly that harness. Evaluating a security agent without an unsafe-action metric in 2026 is like evaluating a self-driving system on "did it arrive?" while ignoring what it hit on the way.

6. Scaling this up: 100K traces and a 3B agent

The 120-episode benchmark answers "how do we grade an investigation?". The follow-up question — "how do we teach one?" — needs orders of magnitude more trajectories. That's the current phase of this project:

• soc-agent-traces-100k / soc-agent-traces-10k — multi-step SOC investigation traces generated by an open, reproducible builder pipeline Space, with the 10K slice curated for training.
• soc-agent-traces-smollm3-3b — a QLoRA fine-tune of SmolLM3-3B on those traces, trained entirely on ZeroGPU via an open trainer Space, with a live investigation demo.

One finding from the broader community makes us deliberately cautious about small agents: the Is it agentic enough? team observed that adding interface surface (more tools, more docs) can help large models while actively hurting small ones, because extra affordances become extra search space. Our depth-decay curve is consistent with that: a 1.5B model masters the mechanical layer and struggles at the composition layer. Whether 100K high-quality traces can push a 3B model's root-cause accuracy past the 0.58 wall — without inflating its hallucinated-evidence rate — is exactly the experiment the trainer pipeline exists to answer, and it will be the subject of the follow-up article.

7. Run it yourself

Everything is Apache 2.0. Evaluate your own agent in ~20 lines:

from datasets import load_dataset
import json

ds = load_dataset("alirezaaminzadeh/soc-agent-bench")
episode = ds["test"][0]

alert = json.loads(episode["initial_alert"])
logs = json.loads(episode["log_corpus"])          # the world
truth = json.loads(episode["hidden_ground_truth"])  # the answer key (keep it hidden!)
forbidden = episode["forbidden_actions"]

# 1. give your agent the alert + the 7 tools over `logs`
# 2. record every tool call and the final incident report
# 3. score the trajectory against gold_evidence / gold_timeline / truth

• Benchmark: soc-agent-bench
• Classical baselines: soc-analyst-baseline
• Tool-calling agent: soc-analyst-tool-use
• Training traces: soc-agent-traces-100k
• Live demo: SOC Agent Traces Space

If you evaluate a model on the benchmark, open a discussion on the dataset repo with your per-metric results — the goal is a community table where root-cause accuracy and unsafe-action rate, not verdict F1, are the headline columns.

8. Takeaways

1. Final-answer metrics saturate and mislead. A bag-of-words model hit 1.00 on severity classification. That number is an indictment of the metric, not evidence of competence.
2. Grade the trajectory. Evidence precision, hallucinated-evidence rate, and unsafe-action rate capture the failure modes that actually block SOC deployment.
3. Depth-decay is the signal to watch. Our 1.5B agent: 0.88 → 0.72 → 0.68 → 0.58 from tool syntax to root cause. Wherever your agent's curve breaks is where your next training data should come from.
4. Safety must be a metric, not a vibe. With tool-poisoning success rates above 60% in the wild, forbidden-action scoring belongs in every security-agent benchmark.

Feedback, criticism, and adversarial re-runs are welcome — the fastest way to improve a benchmark is for someone to break it. Open a discussion on any of the repos above.

References

• Is it agentic enough? Benchmarking open models on your own tooling — Hugging Face
• Model Context Protocol Threat Modeling and Analyzing Vulnerabilities to Prompt Injection with Tool Poisoning
• Securing the Tool Layer: A Threat Taxonomy and Runtime Defense Framework for MCP Deployments — ACL 2026
• Breaking the Protocol: Security Analysis of the MCP Specification
• MITRE ATT&CK