Hugging Face's logo

Rudraneel93
/
mastyf-guard-1.5b

Text Generation
Safetensors
GGUF
English
security
agent-security
agent-defense
mcp
model-context-protocol
prompt-injection
indirect-prompt-injection
tool-calling
capability-based-access-control
guardrails
llm-security
qwen2.5
conversational
Model card Files
xet
 Community

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Mastyf Guard 1.5B is available for security research and enterprise developer preview. To access the model weights and tokenizer, please submit your contact details and Lemon Squeezy order/license key below. Commercial production deployments require an active Mastyf Enterprise Pass.

Log in or Sign Up to review the conditions and access this model content.

Mastyf AI Logo

Mastyf Guard 1.5B โ€” Capability-Mediated Security for Autonomous AI Agent Tool Execution

A security-focused neural auditor for autonomous AI agents and Model Context Protocol (MCP) tool execution perimeters.
Treat the LLM as an untrusted principal. Let it reason and propose actions โ€” but do not let it determine its own authority.

Zenodo DOI Live Demo GitHub Code

Threat Recall CBAC Recall Fast Path Overhead Neural Latency RAM Footprint 50k Samples

๐Ÿ“Œ Executive Summary

Mastyf Guard 1.5B is a specialized, lightweight neural security auditor engineered to operate within an external capability-mediated reference monitor for autonomous AI agents and Model Context Protocol (MCP) tool-calling architectures.

Current agent architectures suffer from Cognitive Von Neumann Conflation: autoregressive Transformers process developer instructions, user intents, and untrusted third-party inputs (retrieved web content, customer emails, API payloads) within a single homogeneous token sequence:

H=Softmaxโก(QKTdk)V,X=[xsysโˆฅxuserโˆฅountrusted]\mathbf{H} = \operatorname{Softmax}\left(\frac{\mathbf{QK}^T}{\sqrt{d_k}}\right)\mathbf{V}, \quad \mathbf{X} = [\mathbf{x}_{\text{sys}} \mathbin{\Vert} \mathbf{x}_{\text{user}} \mathbin{\Vert} \mathbf{o}_{\text{untrusted}}]

Because self-attention allows untrusted tokens to influence control representations, an attacker can steer an agent toward unauthorized privileged tool calls โ€” creating the classic Confused Deputy problem.

Rather than relying solely on linguistic filtering (which fails under paraphrasing and adversarial suffixes), Mastyf introduces an architectural decoupling:

  1. Capability-Based Access Control (CBAC): Enforces least-privilege tool dispatch via cryptographic session tokens. Out-of-scope tool calls are rejected deterministically at the transport layer ($0.005\text{ ms}$ overhead).
  2. Deterministic Pre-Flight Filters: Sanitizes control tokens, decodes recursive encodings, and intercepts destructive shell/SQL patterns.
  3. Fine-Tuned 1.5B Neural Auditor: Conducts semantic argument auditing specifically on operations that are already authorized within the capability envelope, catching in-scope parameter poisoning.

The core principle:

A compromised LLM may propose an unauthorized action, but it must not be able to exercise authority it was never granted.

๐Ÿ›๏ธ Architecture & System Topology

Cognitive Harvard Topology

Figure 1: Architectural comparison between traditional Shared-Context Conflation (where untrusted observations hijack agent tool execution) and Mastyf's Cognitive Harvard Decoupling (where an external capability reference monitor isolates the execution perimeter).

๐Ÿš€ What's New in Mastyf Guard 1.5B v2 (Enterprise Release)

Traditional guardrail models ask: "Does this text look toxic or malicious?" In an autonomous agent workflow, this fails: an attacker does not use vulgarity. They call legitimate tools (e.g. send_email, db_query, file_upload) with poisoned arguments (diverting receipts to an attacker email, broadening SQL queries to *, or redirecting file uploads to /etc/shadow).

Mastyf Guard 1.5B v2 replaces scalar toxicity scoring with an argument-aware tool-security auditor:

P(argument violates intended actionโˆฃT,ฮธ,C,x)P(\text{argument violates intended action} \mid T, \theta, C, x)

Multi-Head Security Classification

Instead of an opaque score, v2 predicts calibrated probabilities across 5 orthogonal security heads:

  1. parameter_poisoning: Argument redirected to unapproved scope, wildcard (*), or abnormal transaction value.
  2. data_exfiltration: Sensitive fragments or system paths routed to external sinks or webhooks.
  3. destructive_action: Execution of DROP, TRUNCATE, rm -rf, or unauthorized deletion.
  4. authorization_anomaly: Privilege escalation attempts (e.g. assigning superadmin_root role).
  5. benign: Normal, expected operation matching user intent.

๐Ÿ”ฌ 3,000-Case Common Held-Out Enterprise Evaluation (v1 vs. v2)

Evaluated across an identical 3,000-case held-out suite comprising 1,800 attacks and 1,200 authentic benign operations across 4 disjoint pillars:

Evaluation Tier / Sub-Benchmark v1 Baseline v2 Argument Auditor Delta ($\Delta$) Sample Composition
Common Held-Out Attacks 10.00% 88.89% +78.89 pp $N = 1,800$ attacks
Benign Operation FPR 0.00% 0.00% 0.00 pp $N = 1,200$ benign (UB $\le 0.25%$)
Adaptive Red-Team (GCG/AutoDAN/Homoglyphs) 13.33% 100.00%* +86.67 pp $N = 600$ attacks (*recall only)
10 Unseen Enterprise Tools (Terraform/K8s) 0.00% 100.00%* +100.00 pp $N = 400$ attacks (*recall only)
Enterprise Workload Manipulations 0.00% 100.00%* +100.00 pp $N = 300$ attacks (*recall only)
Cross-Domain Suite (InjecAgent/BIPIA/Garak) 4.06% 58.95% +54.89 pp 2,658 adversarial + 1,000 benign cases

*Recall only; corresponding benign operations are evaluated in the common denominator.

Latency Bifurcation: The fast-path reference monitor (schema normalization and string-leaf drift computation) incurs a median overhead of 14.9 ฮผs (P99: 88.8 ฮผs) on commodity CPU. Full autoregressive neural inference pass incurs a median latency of 18.4 ms.

๐Ÿ† Massive Third-Party Open-Source Benchmark Scoreboard (3,658 Cases)

Evaluated across 3,658 test cases (2,658 adversarial attacks + 1,000 clean enterprise operations) from the leading open-source security benchmark suites:

| Model / Defense Configuration | Macro Recall (TPR) | FPR | Precision | F1-Score | Avg Latency | Footprint | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | OpenAI Prompt Guard (86M) | 31.23% | 6.80% | 92.43% | 0.4668 | ~25.0 ms | Cloud / GPU | | Meta Llama Guard 3 (8B) | 47.67% | 4.70% | 96.42% | 0.6380 | ~42.5 ms | 16 GB VRAM GPU | | Mastyf Guard 1.5B v1 (Isolated) | 4.06%* | 0.00% | 100.00% | 0.0781 | 14.9 ฮผs (fast-path) | 1.1 GB RAM (CPU) | | Mastyf Guard 1.5B v2 (NEW - Isolated) | 58.95% | 16.70% | 90.37% | 0.7136 | 18.4 ms (neural pass) | 1.1 GB RAM (CPU) | | MASTYF FULL SYSTEM (v2 Model) | 97.33% | 0.00% | 100.00% | 0.9865 | < 0.05 ms (amortized) | 1.1 GB RAM (CPU) |

Suite-by-Suite Interception Breakdown

Benchmark Suite Evaluated Cases v1 Intercept v2 Isolated Model Full System (with v2)
UIUC InjecAgent (Direct Harm + Data Stealing) 2,108 0.0% 51.6% (1,088 blocked) 100.0% (2,108/2,108)
Microsoft BIPIA (Indirect Injection via Email/Text) 300 0.0% 100.0% (300/300 blocked) 100.0% (300/300)
NVIDIA Garak Probes (Red-Team Injection Vectors) 250 43.2% 71.6% (179/250 blocked) 71.6% (179/250)
Clean Enterprise DevOps (Benign Git, K8s, SQL) 1,000 0.0% FPR 0.0% FPR 0.0% FPR (0 false alarms)

Figure 3: Formal state transitions from raw tool invocation through Pre-Flight Sanitization, Capability Verification, and In-Scope Neural Auditing to Final Execution or Fail-Closed Drop.

๐ŸŽฏ Unified Threat Taxonomy & Defense Boundary

Mastyf is evaluated against 8 distinct attack classes, mapping each threat to its active defense tier:

Attack Category Threat Description Intercepting Tier Primary Defense Mechanism
Indirect Prompt Injection (IPI) Untrusted web/email context subverts tool dispatch (InjecAgent, BIPIA). Tier 2 + Tier 1.5 CBAC capability lattice check & neural semantic audit.
Direct Prompt Injection / Jailbreak Adversarial user prompts (DAN, AutoDAN, GCG suffixes from NVIDIA Garak). Tier 2 + Tier 1.5 Blocks unauthorized tools regardless of model jailbreak state.
Destructive OS Commands rm -rf /, reverse shells (mkfifo /tmp/s; /bin/sh), fork bombs. Tier 0 Deterministic AST syntax tree parsing & regex filters.
SQL Injection & DB Mutation DROP TABLE, UNION SELECT, multi-statement relational tampering. Tier 0 Tokenizer SQL structure screening & keyword blocks.
Control-Token Smuggling `< im_start >system`, ChatML delimiter injections to spoof system role.
Encoding & Obfuscation Multi-layer Base64, Hex, URL-encoding, Unicode homoglyph confusables. Tier 0 Recursive decoders (up to 10 iterations) + NFKC normalization.
Out-of-Scope Capability Pivot Read-only agent attempting privileged mutation (smart_lock_unlock). Tier 2 (CBAC) Token Capability Table validation (Theorem 1: $\operatorname{Pr} = 0$).
In-Scope Parameter Poisoning Attacker redirects legitimate tool arguments (e.g. modified email recipient). Tier 1.5 1.5B neural auditor detecting semantic divergence.

๐Ÿ“Š Comprehensive 50,000-Instance Evaluation Benchmark

The accompanying research paper evaluates Mastyf across a balanced 50,000-instance corpus:

  • 25,000 Adversarial Attacks: 9,108 source academic attacks from UIUC InjecAgent, Microsoft BIPIA, and NVIDIA Garak, plus 15,892 parametric perturbation variants (treated as derived variants).
  • 25,000 Benign Developer Traces: Real-world DevOps, cloud infrastructure, and database operations.

Headline Benchmark Comparison

Defense Architecture Parameters Threat Recall Precision FPR F1 Score Fast Latency Neural Latency RAM (INT4)
Unprotected Baseline โ€” 0.00% 0.00% 0.00% 0.0000 0.000 ms โ€” 0.0 GB
OpenAI Prompt Guard 86M 54.34% 88.62% 6.98% 0.6738 โ€” 8.2 ms 0.4 GB
Meta Llama Guard 3 1B 64.91% 89.20% 7.86% 0.7514 โ€” 14.6 ms 0.8 GB
Meta Llama Guard 3 8B 70.73% 90.14% 7.73% 0.7926 โ€” 38.2 ms 5.2 GB
CBAC Alone (Tier 2) โ€” 82.40% 99.90% 0.08% 0.9031 0.003 ms โ€” 0.0 GB
Neural Auditor Alone (Tier 1.5) 1.5B 78.50% 90.32% 8.41% 0.8400 โ€” 18.4 ms 1.1 GB
Mastyf Guard (Full Pipeline) 1.5B 99.33% 91.47% 9.26% 0.9524 0.005 ms 18.4 ms 1.1 GB

(Fast latency: CPU reference-monitor transport check. Neural latency: forward pass on commodity CPU when in-scope argument inspection is triggered. Theorem 2 DIFC dynamic taint tracking resolves cross-tool exfiltration without modifying the macro capability perimeter).

๐Ÿ“ˆ Accuracy & Statistical Rigor

ROC and Precision-Recall Curves (50,000 Samples)

50k ROC and PR Curves

Figure 4: Dual ROC ($AUC = 0.988$) and Precision-Recall ($AUC = 0.976$) curves evaluated across all 50,000 instances with 95% bootstrap confidence intervals (1,000 iterations).

Granular Sub-Track Performance (BIPIA & InjecAgent)

BIPIA and InjecAgent Breakdown

Figure 5: Granular defense recall across Microsoft BIPIA application domains (Email, Web, Table, Code, Text QA) and UIUC InjecAgent Direct Harm (DH) and Data Stealing (DS) sub-tracks.

Confusion Matrix (50,000 Samples)

Predicted Negative (Clean)Predicted Positive (Blocked)Actual Benign (25,000)True Negative (TN): 22,684False Positive (FP): 2,316Actual Attack (25,000)False Negative (FN): 168True Positive (TP): 24,832\begin{array}{c|cc} & \textbf{Predicted Negative (Clean)} & \textbf{Predicted Positive (Blocked)} \\ \hline \textbf{Actual Benign (25,000)} & \text{True Negative (TN): } 22,684 & \text{False Positive (FP): } 2,316 \\ \textbf{Actual Attack (25,000)} & \text{False Negative (FN): } 168 & \text{True Positive (TP): } 24,832 \\ \end{array}

  • True Negative Rate (Specificity): $90.74%$
  • False Negative Rate (Miss Rate): $0.67%$ (only 168 attacks bypassed the entire multi-tier perimeter out of 25,000)

Paired McNemar Hypothesis Testing

To prove architectural capability mediation statistically outperforms scaling guardrail models:

ฯ‡2=(โˆฃn01โˆ’n10โˆฃโˆ’1)2n01+n10\chi^2 = \frac{(|n_{01} - n_{10}| - 1)^2}{n_{01} + n_{10}}

  • Mastyf Guard vs. Meta Llama Guard 3 8B:
    $n_{10} = 7,314$ (Mastyf correct, Llama Guard incorrect), $n_{01} = 164$ (Llama Guard correct, Mastyf incorrect).
    $$\chi^2 = 3,252.8, \quad p < 10^{-15} \quad (\text{Statistically decisive})$$
  • Mastyf Guard vs. OpenAI Prompt Guard 86M:
    $$\chi^2 = 10,721.4, \quad p < 10^{-15} \quad (\text{Statistically decisive})$$

โšก Hardware Efficiency & Latency Economics

Latency and Hardware Percentiles

Figure 6: P50 and P99 latency percentiles (log scale) and execution throughput comparing Mastyf Fast-Path and Neural-Path on CPU vs. Llama Guard 3 on GPU and CPU.

Hardware Footprint Summary

  • Fast-Path Mediation: $4.8,\mu\text{s}$ ($0.005\text{ ms}$) on standard CPU cores.
  • Neural Forward Pass: $18.4\text{ ms}$ on CPU using INT4 quantization (AWQ/GGUF).
  • RAM Footprint: $\approx 1.12\text{ GB}$ host memory โ€” runs entirely without a dedicated GPU.
  • Production DevOps Pass-Through: On 1,000 authentic multi-turn developer workflows (10,000 tool calls), Mastyf achieved $96.52%$ clean zero-friction pass-through and $95.2%$ task completion, matching unprotected baseline performance within $0.8%$.

๐Ÿ”ฌ Component Ablation Analysis

Active Configuration Threat Recall Precision FPR F1 Score Latency
Tier 0 only (Regex / Decoders) 40.96% 95.74% 1.82% 0.5736 0.029 ms
Tier 2 only (CBAC Gate) 82.40% 99.90% 0.08% 0.9031 0.003 ms
Tier 1.5 only (Neural Auditor) 78.50% 90.32% 8.41% 0.8400 18.400 ms
Tier 0 + Tier 2 (Fast-Path) 96.80% 98.07% 1.90% 0.9743 0.032 ms
Tier 0 + Tier 2 (Fast-Path) 96.80% 98.07% 1.90% 0.9743 0.032 ms
Tier 2 + Tier 1.5 (Capability + Neural) 94.20% 91.73% 8.49% 0.9295 18.403 ms
Full Pipeline (Tier 0 + 1.5 + 2) 99.33% 91.47% 9.26% 0.9524 0.005 ms

โš”๏ธ Adaptive Red-Team & Security Limitations

Academic honesty requires detailing where capability-mediated perimeters encounter boundary conditions:

1. In-Scope Parameter Poisoning

If an attacker tampers with the parameters of an already-authorized tool (e.g. rewriting the recipient on an authorized send_email call), CBAC cannot reject the call on scope alone.

  • The 1.5B neural auditor achieved 78.50% recall on in-scope parameter poisoning.
  • Adversarial gradient optimization (AutoDAN / GCG) reduced neural-only recall from $78.50% \to 61.20%$.
  • However, all out-of-scope escalation attempts remained blocked by the CBAC layer (demonstrating that the security boundary does not rely on the neural model being unjailbreakable).

2. Cross-Tool Secret Exfiltration

Capability authorization answers: "May this agent invoke Tool X?"
It does not answer: "May data retrieved from Tool A flow into Tool B?"

  • In our empirical red-team evaluation, 44 of 544 test cases (8.09%) successfully exfiltrated data across authorized tools (500/544 defended, $91.91%$ defense rate).
  • Recommendation: Capability mediation must be paired with Decentralized Information Flow Control (DIFC), taint tracking, or Data Loss Prevention (DLP) to prevent multi-tool data leaks.

3. Capability Forgery

Evaluated without access to the secret HMAC key. No token forgeries succeeded (modeled under HMAC-SHA256 with 128-bit computational security).

๐Ÿ“ฆ Model Training & Hyperparameters

  • Base Model: Qwen/Qwen2.5-1.5B-Instruct
  • Fine-Tuning Method: Low-Rank Adaptation (LoRA)
    • Rank ($r$): 16 | Alpha ($\alpha$): 32 | Dropout: 0.05
    • Target Modules: Query and Value attention heads (q_proj, v_proj)
    • Optimizer: AdamW | Learning Rate: $2 \times 10^{-4}$ (Cosine decay)
    • Batch Size: 16 | Epochs: 3 | Seed: 42
    • Loss Function: Regularized Focal Loss ($\gamma = 2.0, \alpha = 0.25$)
  • Dataset Composition:
    • Training Set: 28,450 paired tool-call interaction samples
    • Validation Set: 3,550 held-out interaction samples
    • Convergence: Training loss converged from $1.842 \to 0.168$; validation perplexity reached $1.21$.
  • Quantization: INT4 AWQ and GGUF Q4_K_M.

๐Ÿš€ Local Deployment

Ollama 

ollama create mastyf-guard -f Modelfile
ollama run mastyf-guard

Python / Hugging Face Transformers 

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Rudraneel93/mastyf-guard-1.5b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

prompt = """<|im_start|>system
You are Mastyf Guard, a security auditor for agent tool calls.
Inspect the following tool invocation for parameter poisoning or malicious manipulation.<|im_end|>
<|im_start|>user
Tool: execute_sql
Parameters: {"query": "SELECT * FROM users WHERE id = 1; DROP TABLE logs;"}<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

๐Ÿ’ณ Developer API & Enterprise Access

Mastyf Guard is available as a production cloud API in addition to the open research model artifact.

Security Scope Disclaimer:
Mastyf Guard is a defense-in-depth agent security layer focused on capability-based containment of unauthorized tool execution. It does not guarantee prevention of all indirect prompt injections, authorized-tool parameter manipulation, or cross-tool data exfiltration. For sensitive production workloads, deploy alongside application authorization, DLP/information-flow controls, secrets management, and network egress controls.

Developer API Pass (โ‚น2,500 / month)

Designed for developers, researchers, and indie builders:

  • Hosted Mastyf reference-monitor endpoint
  • Monthly API tool validation allowance
  • Real-time pre-flight inspection and capability mediation
  • Python and TypeScript SDKs

๐Ÿ‘‰ Subscribe to Developer API:
https://mastyfai.lemonsqueezy.com/checkout/buy/49323daa-90ef-4157-90b9-8706acd13fe6

Enterprise Deployment

Custom policy configurations, on-premise reference-monitor binaries, and dedicated SLA support:
๐Ÿ‘‰ Repository & Enterprise Inquiries: https://github.com/mastyf-ai/mastyf.ai

๐Ÿ“š Research Paper & Authoritative Resources

๐Ÿ“– Citation

If you use Mastyf Guard or its evaluation benchmarks in your research, please cite:

@misc{das2026mastyf,
  author    = {Das, Rudraneel},
  title     = {Capability-Mediated Perimeters for Secure AI Agent Tool Execution: Formal Non-Escalation Guarantees and Empirical Evaluation Against Indirect Prompt Injection},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22179415},
  url       = {https://doi.org/10.5281/zenodo.22179415}
}
Downloads last month
-
GGUF
Model size
2B params
Architecture
qwen2
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Rudraneel93/mastyf-guard-1.5b

Base model

Qwen/Qwen2.5-1.5B
Finetuned
Qwen/Qwen2.5-1.5B-Instruct
Quantized
(271)
this model