Reading list

Papers, investigations, and ideas worth keeping, with notes on why they matter.

Sources & reading notes

All collections ↑

92 sources in 6 collections

ML supply chain & steganography

19 sources

Work on model artifacts, weight-space behavior, and the trust decisions around a model supply chain.

2026-09

Backdoor in the Loop: Compromising Agentic Search via Malicious Retrievers

Xu, Hua & Xiao

A malicious retriever checkpoint can steer an otherwise unchanged search agent by repeatedly placing an attacker-chosen, existing document in its top results. In tests with two retrievers and two question-answering datasets, the target appeared in the top three for 99.8–100 percent of tested questions and drove extra search rounds. A decoy that could later be removed also weakened the tested detectors. The attacker controls the retrieval model, not the corpus or agent; the result is a local benchmark finding, not a measured production compromise.

arxiv.org · September 2026
2026-09

Backdoors in Learning-Based Industrial Robotic Arm Manipulation: An Empirical Security Study

Zhang, Zeng, Gu & Pisharody

Poisoning five demonstrations, 10 percent of the training set, made a visuomotor policy take attacker-chosen physical actions when a visual trigger appeared. On the tested FANUC and xArm tasks, the wrong action occurred in 8–10 of 10 triggered rollouts. An off-task visual sanitization step restored nominal behavior in one small comparison, but assumes this kind of trigger. The evidence is compelling for the tested policy and tasks, not a general result for all robot or vision-language systems.

arxiv.org · September 2026
2026-09

High-Capacity Robust Medical Image Exfiltration via Neural Network Weight Replacement

Thellier, Li, Ayache & Delingette

A researcher who can train and export a model, but cannot export patient scans, can encode images into its weights and reconstruct them after export. The paper fits up to 99 brain MRI volumes into a functional 30 MB model; the recovered images are approximate but anatomically recognizable. Fine-tuning, pruning, and quantization did not reliably erase the channel in these experiments. It is a strong reason to treat model export as a data-export decision, while keeping the claim short of demonstrated patient re-identification.

MICCAI 2026 · September 2026
2026-09

Permutation-Based Stegomalware in Large Language Models: Threats and Countermeasures

Danny Wood & James Stringer

Behavior-preserving permutations of model weights serve both sides of this problem: an attacker can encode a payload in the permutation, while a defender can re-permute all parameters to disrupt extraction with little measured quality loss. The attack still needs a separate loader to execute anything, and the defense assumes the model can be safely loaded and its symmetries identified. That distinction between hidden bytes, extraction, and execution makes this a useful companion to EvilModel.

arxiv.org · September 2026
2026-08

Beyond Source: An Empirical Study of Python Bytecode Security Risks

Chen, Xie & Li

Python runtimes execute bytecode directly through .pyc files, compiled-only modules, and marshalled code objects, which opens an inspection-execution gap that source-centric package security misses entirely. Across 1,034,843 PyPI artifacts they found 7,388 containing bytecode, including 28,193 source-less .pyc files local to an artifact. The tooling is not robust either: real and adversarially mutated bytecode drove decompilers into managed exceptions, timeouts, and native process failures. The same shape as the model-format problem one ecosystem over, since what the scanner can read is not what the runtime will execute.

arxiv.org · August 2026
2026-08

Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs

Lin, Jin, Liu, Wu & Yu

Existing VLM backdoors bind fixed triggers to a fixed target set at poisoning time. This one implants a programmable backdoor from a single poisoning phase, so the attacker picks previously unseen target-caption semantics at inference and synthesizes a matching stealthy trigger on demand. The mechanism is a feature-space trigger steganography that maps any specified caption to a visual trigger, paired with a poisoning strategy that teaches trigger-as-instruction rather than a memorized pattern. Decoupling target selection from poisoning is what turns a backdoor into a channel.

arxiv.org · August 2026
2026-08

LoRAScan: Detecting Backdoor Prompts in Low-Rank Adapters via Down-Projection Activation Spikes

Obidov, Yu, Guo & Yang

Adapter-agnostic defenses merge the adapter into the base model first, which dilutes exactly the signal they are hunting for. LoRAScan stays adapter-aware and rejects trigger-bearing inputs at inference time from activation spikes in the down-projection, rather than condemning the whole adapter and requiring separate mitigation. Worth reading beside the weight-space LoRA detector: one screens the artifact before use, this one screens the traffic during it.

arxiv.org · August 2026
2026-07

ShadowPickle: Evading Machine Learning Model Scanners via Stealthy Pickle Deserialization Attacks

Pradhan, Nambiar & Soremekun

Three pickle deserialization attacks that abuse the Pickle VM external module import mechanism to execute payloads during load. The Overwritten variant evades 63 percent of scanners, up to 50 percent better than prior attacks, across ten state-of-the-art scanners and four model hosting platforms. Read this against any argument that scanning solves pickle: the format keeps the execution path, so the scanner is left guessing at intent.

arxiv.org · July 2026
2026-07

Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering

Briglia, Maljkovic, Cina, Oneto, Masi & Roli

A trigger-gated additive modification of an intermediate representation, dormant by default and producing zero change on clean inputs, carried in the shared model artifact rather than in the learned weights. One implant compromises visual question answering, text-to-image, retrieval, and ranking fairness at once. The framing is the part worth keeping: a VLM deployment inherits executable behavior encoded in the artifact, not just parameters, so the proposed audit reads the distributed logic instead of the weights.

arxiv.org · July 2026
2026-05

Locking Pretrained Weights via Deep Low-Rank Residual Distillation

Sakamoto, Ablin, Danieli & Cuturi

DLR-Lock replaces each pretrained MLP with a deep low-rank residual network that increases activation memory during fine-tuning and creates architectural mismatches that frustrate standard optimization. Tested against adaptive attackers with full knowledge of the defense. Framing the goal as locking rather than watermarking shifts from post-hoc detection to resistance, which is a different threat model.

arxiv.org · May 2026
2026-05

BCJR-QAT: A Differentiable Relaxation of Trellis-Coded Weight Quantization

Venugopalan Iyengar

Trellis-coded quantization encodes model weights using error-correcting code structure; the Viterbi decoding step is non-differentiable, which breaks gradient flow during QAT. BCJR-QAT replaces Viterbi with the forward-backward sum-product algorithm to restore end-to-end differentiability. Trellis-coded weight encoding is the quantization scheme most directly analogous to steganographic embedding; this advances its training-side infrastructure.

arxiv.org · May 2026
2026-05

Undetectable Backdoors in Model Parameters: Hiding Sparse Secrets in High Dimensions

Choudhary, Patlan, Palumbo, Hooda, Fawaz & Jha

Sparse Backdoor hides a trigger-gated perturbation in image-classifier weights beneath Gaussian dither. Under a stated margin condition and Sparse-PCA hardness assumption, distinguishing those weights from a particular dithered clean reference is at least as hard as Sparse PCA. That is a meaningful limit on weight-only detection in the paper's game, not a proof that every behavioral audit fails.

arxiv.org · May 2026
2026-04

Provably Secure Steganography Based on List Decoding

Pang & Bai (Tsinghua University)

Pushes provably-secure linguistic steganography toward higher embedding capacity by maintaining a list of candidate decodings rather than a single one. Directly relevant to the entropy-budget question in any LLM-mediated covert-channel design.

arxiv.org · April 2026
2026-04

Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggers (BadStyle)

Wei et al.

Style-level (not token- or syntax-level) backdoor triggers, generated by an LLM as a poisoned-sample synthesizer. Adds an auxiliary target loss to stabilize payload injection during fine-tuning. Evaluated against seven model families.

arxiv.org · April 2026
2026-04

CSC: Turning the Adversary's Poison against Itself

Shi, Guo, Chen, Zhu, Liu & Zhou

Poisoned samples form isolated latent-space clusters early in training because triggers dominate their feature representation. CSC exploits this: cluster, segregate, relabel to a virtual class, fine-tune. Near-zero ASR across four datasets and twelve attack variants with minimal clean accuracy loss. The trigger design that makes poisoned samples distinctive to the model also makes them distinctive to an auditor.

arxiv.org · April 2026
2026-02

Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise

Fogel, Hofman, Cohen & Vainshtein

Chat templates are Jinja2 programs that run on every inference call, which makes them a code channel that ships alongside the weights without touching them. Triggered backdoors dropped factual accuracy from 90 percent to 15 percent across eighteen models and hijacked tool use across 3,868 agent episodes, and the poisoned artifacts passed every scan on the largest open model distribution platform. Same attack surface as the Llama-Drama template work, aimed at distribution rather than at a parser bug.

arxiv.org · February 2026
2026-02

Weight space Detection of Backdoors in LoRA Adapters

Puertolas Merenciano, Vasyagina, Zhu, Ferrando & Chaudhary

Pulls five spectral statistics from the low-rank updates across the Q, K, V, and O projections, giving a 20-dimensional signature per adapter, then trains a logistic regression on it. 100 percent accuracy on unseen adapters across Llama-3.2-3B, Qwen2.5-3B, and Gemma-2-2B, spanning instruction-following, reasoning, QA, code, and classification. Detection that needs neither the trigger nor a forward pass is exactly the property a hub needs, since it cannot execute everything it stores.

arxiv.org · February 2026
2021

EvilModel: Hiding Malware Inside of Neural Network Models

Wang, Liu, Cui

The canonical reference for byte-level steganography in float32 weight tensors. Explicitly defers the decoder to a separately-deployed loader, which is the substantive limitation when read against a channel/decoder/substrate framework.

arxiv.org · 2021
2017

BadNets: Identifying Vulnerabilities in the ML Supply Chain

Gu, Dolan-Gavitt, Garg

Origin paper for trigger-based co-trained backdoors. The decoder and the channel are baked into the network's weights together, which is why detection has to be behavioral rather than static.

arxiv.org · 2017

LLM red-teaming & jailbreaks

19 sources

Research on model behavior under adversarial inputs and the methods used to evaluate it.

2026-10

Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models

Hu et al.

Instead of searching for one reusable prompt, Red-TTT updates a small attacker adapter for each harmful behavior and victim model. In single-turn HarmBench tests against four models at 120 target queries per behavior, its mean classifier-judged success was 72.4 percent against 55.9 percent for a query-matched Best-of-N baseline; a separate judge preserved the ordering. The target-call budget is comparable, but the attacker's local training cost is additional, and these results do not establish multi-turn or tool-use success.

arxiv.org · October 2026
2026-10

MLCommons Jailbreak Benchmark v1.0

Maple et al. (MLCommons)

A paired baseline and adversarial test of single-turn text jailbreaks across eight open-weight systems, 264 seed prompts, and eleven hazard categories. Unsafe responses rose from 11.08 to 18.65 percent, a 7.57-point average resilience gap. The more useful contribution is the evaluation discipline: it calibrates automated grading against human labels and reports measurement error, so a jailbreak score has an explicit baseline and a known scope. It says nothing about multi-turn agents or tool execution.

arxiv.org · October 2026
2026-09

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

Animesh Shaw

One evaluation harness counted the choice of a tool as attacker success without checking whether its arguments carried the malicious action. Rescoring the same 258 traces cut apparent success from 21.7 to 1.2 percent; a separate delivery defect silently dropped 91 percent of injected payloads. The replacement checks delivery, task capability, trace, and final effect. These are measurements of one simulated five-tool suite, not a new ranking of frontier defenses, but they show how a broken harness can manufacture a result.

arxiv.org · September 2026
2026-09

An Experimental Evaluation of Multimodal Prompt Injection Attacks on Agentic AI Frameworks

Nguyen & Husain

MMPIBench places attacker instructions in images read by agents and traces the result from perception through planning to a tool call. Across 720 visual runs, 92 were judged to attempt compliance but only eight completed the attacker objective; the main break came when models recognized the instruction and declined to act. Its smaller audio study reached far fewer model-framework combinations and used a different task framing, so the higher audio completion rate is not a like-for-like comparison. The value is separating exposure, attempted compliance, tool use, and final effect.

arxiv.org · September 2026
2026-08

Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

He, Wen, Qi, Zhou, Yu, Li, Lu & Zhang

Attack success rate depends on how many target calls an attack is allowed, and most studies report it without holding that budget constant, so the comparisons are not comparable. Fair-ASR fixes a shared target-call budget and tracks attacker calls separately for efficiency. Re-evaluating 11 representative attacks, rankings change substantially across budgets, simple stochastic perturbations and hand-crafted templates stay highly competitive under equal target access, and none of the evaluated LLM-driven methods was efficient in both dimensions. A headline number that hides its budget is the same disease as an injection rate that hides its decomposition.

arxiv.org · August 2026
2026-08

The Anatomy of a Prompt Injection: A Component Model for Structured Analysis

Jeremy McHugh

Four years on, prompt injections are still documented as verbatim strings, which makes them hard to compare, label, or mutate without fragile string matching. This proposes a seven-component model, carrier, delivery vector, concealment, context-break, privilege escalation, payload, and return channel, as five artifact fields plus two environment fields. Because models compile many surface wordings into the same executable action, the labels have to track attacker intent, tool targets, and sinks instead of the wording. Naming the parts is what lets you argue about coverage rather than trade examples.

arxiv.org · August 2026
2026-06

PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections

He, Miculicich, Sharma, Fox, Lee, Tang, Pfister & Le

Generates realistic source-aware test cases for indirect prompt injection and evolves them through feedback-driven exploration, instead of sampling from a fixed library of known payloads. Localizes which injected source caused the failure rather than only reporting that the agent failed. It stays effective with defenses deployed, which is the evaluation condition most red-teaming papers quietly skip.

arxiv.org · June 2026
2026-06

NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms

Lee, Choi, Kim, Park & Kim

Puts LLM operator agents in a simulated nuclear power plant control room and runs adaptive multi-turn attacks against the operator team. Between 8.7 and 12.1 percent of attack sessions across four models ended in loss of a critical safety function, and the vulnerabilities were distinct per model rather than overlapping. The transferable part is the harness rather than the scenario: single-turn refusal rates say almost nothing about how an agent holds up over a session.

arxiv.org · June 2026
2026-05

Jailbroken Frontier Models Retain Their Capabilities

Zhu, Wang, Bao & Wei

28 jailbreaks across five benchmarks, Claude models from Haiku 4.5 to Opus 4.6. The capability tax scales inversely with model strength: Haiku 4.5 loses 33.1% average capability under jailbreaks, Opus 4.6 loses 7.7%. The most sophisticated attacks approach zero degradation. Safety assessments that rely on capability loss as a self-limiting mechanism are working from a false premise.

arxiv.org · May 2026
2026-05

Jailbreaking Vision-Language Models Through the Visual Modality

Azulay, Dubiński, Li, Mittal & Gandelsman

Four attack methods targeting the visual processing path of VLMs: encoding harmful instructions as visual symbols with a legend, object substitution, text replacement in images, and visual puzzles. A visual cipher achieved 40.9% ASR on Claude-Haiku-4.5 vs. 10.7% for the text equivalent. The gap quantifies how little text-focused safety training transfers to vision.

arxiv.org · May 2026
2026-05

Guaranteed Jailbreaking Defense via Disrupt-and-Rectify Smoothing

Lin, Niu, Ji & Gao

DR-Smoothing adds a two-stage prompt processing scheme: disrupt the input, then rectify it back toward a normal distribution before passing to the model. Prior disrupt-only methods left the model seeing out-of-distribution prompts. The rectification step removes that problem while preserving disruption as the defense layer. Includes theoretical bounds on defense success probability.

arxiv.org · May 2026
2026-05

Re-Triggering Safeguards within LLMs for Jailbreak Detection

Lin, Niu, Ji, Huang & Gao

Rather than training a standalone jailbreak classifier, this uses embedding disruption to re-activate the model's own internal safeguards for detection. The defense reuses what the model already knows. Effective against adaptive attacks in evaluation. Same authors as DR-Smoothing; this paper covers detection, DR-Smoothing covers the defense-side response.

arxiv.org · May 2026
2026-05

CALYREX: Cross-Attention Layer Extended Transformers for System Prompt Anchoring

Li Lixing

Adds cross-attention at the final eighth of transformer layers to give system prompts a structurally distinct processing path rather than treating them identically to user input. At 8B scale: +7.4% on IFEval, +16.3% on multi-turn instruction adherence, -13% many-shot jailbreaking ASR. The architectural approach rather than a tuning one is what makes the gains scale-consistent.

arxiv.org · May 2026
2026-05

Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization

Zhou, Zhao, Zhong, Liang, Chen et al.

Recasts jailbreaking as inference-time policy optimization in an adversarial decision process. A self-evolving metacognitive loop diagnoses the target's defense logic and refines the attack trajectory through structured feedback. 89.2% average ASR across 10 models including 76.0% on O1 and 78.0% on GPT-5-chat, at 8.2x lower compute than prior methods.

arxiv.org · May 2026
2026-05

LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

Zhang, Yang, Jiang, Zhang, Zhao et al.

819 test cases for evaluating safety in LLM agents operating in actual operating systems. Identifies Execution Hallucination: agents verbally refuse a request while the harmful OS-level action completes undetected. Running in real OS environments rather than simulation is what exposes this gap between stated refusal and actual execution.

arxiv.org · May 2026
2026-05

When Prompts Become Payloads: SQL Injection via LLM-Driven Natural Language Interfaces

Motlagh, Hajizadeh, Majd, Najafi, Cheng & Meinel

Conversational interfaces that translate natural language to SQL queries inherit SQL injection as a threat class. The proposed defense stacks three layers: front-end prompt sanitization, a behavioral/semantic anomaly detector, and a signature layer for known patterns. The LLM translation layer that creates the vulnerability is also what makes the sanitization non-trivial.

arxiv.org · May 2026
2026-04

Adaptive Instruction Composition for Automated LLM Red-Teaming

Zymet et al. (Capital One AI Foundations)

Replaces random combination of crowdsourced jailbreak ingredients with a contextual-bandit learner that scores combinations based on prior success. Roughly 2,200-parameter bandit on top of SBERT embeddings. Transfers across models without retraining.

arxiv.org · April 2026
2026-04

Transient Turn Injection: Stateless Multi-Turn Vulnerabilities

Rayhan & Jahan

Distributes adversarial intent across stateless turns, evading moderation that evaluates each turn independently. Notable for showing that the threat model "single-turn safety classifier" is incomplete against an attacker LLM operating across sessions.

arxiv.org · April 2026
2024

Persistent Pre-Training Poisoning of LLMs

Zhang, Rando, Evtimov, Carlini, Tramèr et al. (Meta · ETH Zürich · CMU · Google DeepMind)

Poisoning 0.1% of pre-training data is enough for three of four backdoor objectives (DoS, belief manipulation, jailbreaking) to survive post-training. DoS persists at 0.001%. The supply-chain layer the threat model has to start at.

arxiv.org · 2024

MCP & agentic security

20 sources

Studies of agent tools, workspace context, and the boundaries between connected services.

2026-10

How We Hijacked an AI Agent With a Single Email

Salt Labs Research Team

Salt placed instructions in an email that a Manus agent read while doing a legitimate task. In the researchers' test sandbox, the agent ran obfuscated JavaScript through Node before a warning appeared, and the payload opened a reverse shell. Connected-service tokens were present in the environment, which raises the stakes, but the report does not separately demonstrate access to those accounts through the shell. The case turns an untrusted email body into command authority; Salt says the issue was resolved.

salt.security · October 2026
2026-10

Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment

Xie, Luo, Liu et al.

Across six real coding agents, the researchers found a gap between refusing to reveal hidden instructions in chat and copying them into structured tool-call arguments. They then combined malicious tool descriptions and tool returns to induce code execution in the tested setups. The result depends on the agent, model, and tool configuration, but the boundary is general: tool metadata and output are untrusted input, and a plausible-looking call still needs authority checks at the action boundary.

ISSTA 2026 · October 2026
2026-09

SkillSecurer: Detecting and Patching Prompt-Injection Vulnerabilities in AI Agent Skills

Mecca, Verna, Bouchari, Jha & Mellia

SkillSecurer detects and localizes instructions hidden in agent skills: it found all 345 controlled injections in the authors' test. Scanning 954 popular skills flagged 168, but a manual sample confirmed 84 latent risks among 100 findings, so the flag rate is not an exploit rate. In live tests of five selected skills, harmful side effects appeared in 13 of 30 runs before patching and 3 of 30 after. The execution results are useful, while the small selected sample limits what they say about the wider ecosystem.

arxiv.org · September 2026
2026-09

The Shared Clipboard Inside the Sandbox: Cross-Account Data Leakage in ChatGPT

Alexey Bukhteyev (Check Point Research)

An attacker-controlled shared chat or custom GPT carried a hidden task that made a victim's ChatGPT session read connected Gmail data and place it in a shared internal service the attacker could access. The visible response did not show the transfer, which makes the case more than a bad answer: cross-account mutable state bypassed the intended account boundary. OpenAI decommissioned the shared service before the report was completed; read this as a failure mode for state isolation and connector privilege, not a claim of an unfixed live exposure.

research.checkpoint.com · September 2026
2026-08

Exposed by Design: A Dynamic Security Assessment of Internet-Facing MCP Servers at Scale

Nicolas Padilla

Passive discovery across eleven sources plus active testing with 34 modules covering 10 MCP-specific vulnerability classes. Of 640 confirmed production servers, 414 were dynamically audited and yielded 68 reportable vulnerabilities including SQL injection, SSRF against cloud metadata services, prompt template injection, and path traversal via cursor manipulation. 91.8 percent lacked OAuth authentication and 687 tool instances exposed shell execution with no access controls. The quieter finding is the ecosystem one: 41.6 percent of confirmed servers disappeared within three days.

arxiv.org · August 2026
2026-08

ColluSkill: Adversarial Cross-Skill Composition for Evading Agent Skill Scanners

Zeng, Qin, Li, Jia, Liu & Jia

Skill scanners inspect one skill at a time, so an intent split across several individually plausible skills passes every check and only becomes harmful in composition. ColluSkill decomposes a payload into interdependent sub-skills joined by contextual dependencies, artifact passing, and execution handoffs, then uses scanner feedback to file down the suspicious signal in each piece. ChainGuard is the paired defense that reasons over the chain. Choosing the wrong unit of analysis is the same failure ShadowPickle exploits in model scanners.

arxiv.org · August 2026
2026-08

Workspace Topology as an Attack Vector in Agentic Coding Assistants

Day, Yadlapalli, Venkatapathy et al. (Cognizant AI Lab)

Treats the shape of the workspace as the independent variable: directory depth, codebase modularity, in-file injection position, and context framing, measured against indirect prompt injection success across open-source repositories spanning 10 languages and 6 engineering domains. Modularity moves the attack success rate significantly, with highly modular environments faring better. A good complement to the sandbox-escape work: that asks what an agent can reach, this asks how the layout of what it reads changes the odds.

arxiv.org · August 2026
2026-07

The Week of Sandbox Escapes

Cohen, Lisichkin & Fogel (Pillar Security)

Four repeatable failure modes across Cursor, Codex, Gemini CLI, and Antigravity: denylist sandboxes that cannot enumerate an operating system, project configuration files that are really executable code, allowlists that trust a command name rather than the actual invocation, and privileged local daemons sitting outside the boundary entirely. In nearly every case the agent never broke the sandbox, it wrote something a trusted component outside later ran. The executable-config mode is the Amazon Q pattern generalized across four tools.

pillar.security · July 2026
2026-06

MCP Auto-Execution: From Git Clone to Cloud Compromise in Amazon Q VS Code Extension

Maor Dokhanian (Wiz)

The extension loaded MCP server configs from the workspace root with no consent prompt and no workspace trust check, and the servers it spawned inherited the full user environment. Opening a cloned repository was the entire exploit chain, straight through to AWS credentials, cloud CLI tokens, and the SSH agent. CVE-2026-12957, remediated in language server 1.65.0. Workspace files are untrusted input, and every agentic IDE that auto-loads tool configuration is making the same bet this one lost.

wiz.io · June 2026
2026-05

Evaluating Tool Cloning in Agentic-AI Ecosystems

Kim, Jiang, Hu, Jia & Gong

8,861 repositories, 100,011 tools from MCP and Skills platforms. 60% of high-Jaccard candidates and 85% of high-ssdeep candidates are manually verified clones. Vulnerable code in a cloned tool propagates to all downstream repositories automatically. Benchmark splits built from "diverse" tool datasets may be evaluating heavily duplicated code. A supply-chain amplification pattern that pre-dates LLMs.

arxiv.org · May 2026
2026-05

Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning

Kereopa-Yorke, Diaz, Wright, Johnston, Del Rosario & Lynar

Differs from prompt injection by targeting the knowledge graph data agents reason over rather than their instructions. Six attack scenarios against a production knowledge graph with 42M nodes across nine models from three providers. All models accepted fabricated security claims at 100% under directed queries. GPT-5.1 showed 0% trust in inline evaluation but 100% under actual tool-use, which is the key finding: the delivery channel changes the attack surface.

arxiv.org · May 2026
2026-05

Self-Adaptive Multi-Agent LLM-Based Security Pattern Selection for IoT Systems

Jamshidi, Khomh, Fung & Nafi

ASPO integrates LLM reasoning with deterministic enforcement inside a MAPE-K control loop. LLM agents propose mitigations; an optimization engine ensures proposals are conflict-free and resource-feasible before acting. 100% conflict-free activation on a 500-1000 decision testbed. The separation between reasoning and enforcement is the design principle worth generalizing.

arxiv.org · May 2026
2026-05

Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents

Chunxiao Wang

Black-box drift detection using cosine similarity between user prompts and behavioral anchor texts, aggregated by weighted top-k mean over BGE-m3 embeddings. ROC AUC 0.83 on real session traces. Available as a Claude Code plugin and MCP server with Merkle-chained audit logging. The ~30-point gap below white-box methods is the explicit cost of not touching model weights.

arxiv.org · May 2026
2026-04

MCP Pitfall Lab: Developer Pitfalls in MCP Tool Server Security

Hao & Tan

Six-class pitfall taxonomy (P1–P6) split into statically-checkable (Tier-1) and trace/dataflow-dependent (Tier-2) classes. Three workflow challenges (email, document, crypto) with hardened-vs-baseline server pairs and three attack families: tool-metadata poisoning, puppet servers, image-to-tool chains.

arxiv.org · April 2026
2026-04

CAP: Controllable Alignment Prompting for Unlearning in LLMs

Wang, Guo, Pu, Pu, Yang et al.

Machine unlearning without modifying model parameters: a prompt generator trained via reinforcement learning collaborates with the LLM to suppress target knowledge while preserving general capabilities. Works on closed-source models. Reversal is possible by revoking the prompt rather than retraining, which matters for unlearning as a compliance tool when legal hold periods end.

arxiv.org · April 2026
2026-04

Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain

Liu, Shou, Wen, Chen, Fang & Feng

Third-party API routers sit between the agent and the model with full plaintext access to every in-flight JSON payload and no integrity protection on either side. Measured across 428 routers: 9 actively injecting code, 2 using evasion triggers, 17 reaching for the researchers' AWS credentials, and 1 draining a test wallet. The router is an unexamined hop in the supply chain, and this is the paper that actually counts what is happening there.

arxiv.org · April 2026
2026-04

Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems

Qu, Liu, Geng, Deng, Li, Zhang, Zhang & Ma

DDIPE hides instructions inside the code examples and configuration templates that ship in skill documentation, so the payload fires when the agent reuses that material rather than needing a crafted prompt. 1,070 adversarial skills across 15 MITRE ATT&CK categories, bypass rates of 11.6 to 33.5 percent over four frameworks and five models, and only 2.5 percent evading both static detection and alignment. Explicit instruction attacks scored zero against strong defenses, which is the contrast: the implicit path is the one that works.

arxiv.org · April 2026
2026-02

Threat modeling and prompt injection in Comet

Trail of Bits

ML-centered threat modeling applied to an agentic browser. Four prompt-injection techniques against the AI assistant, all chained to exfiltrate Gmail data. The methodology, TRAIL, is more transferable than any individual finding.

blog.trailofbits.com · February 2026
2025-07

Trivial Trojans: Cross-Tool Exfiltration via Minimal MCP Servers

Croce & South

Concrete demonstration of cross-server data exfiltration in MCP. The barrier-to-entry argument matters: this is not a sophisticated attack class, which is the point.

arxiv.org · July 2025
2025

Beyond the Protocol: Attack Vectors in the MCP Ecosystem

Song et al.

First end-to-end empirical evaluation of attacks against MCP. Four attack categories: tool poisoning, puppet attacks, rug pull, and exploitation via malicious external resources. Useful as the lay-of-the-land paper before any MCP-specific work.

arxiv.org · 2025

Inference infrastructure & ML platforms

16 sources

Research on the services, shared resources, and data flows around deployed models.

2026-10

LMCache: Unauthenticated Remote Code Execution via Pickle Deserialization

Yuval Moravchick (JFrog)

In LMCache multiprocess mode, a reachable ZeroMQ transport accepts an unauthenticated message whose MessagePack extension is unpickled while decoding arguments, before the requested handler runs. JFrog demonstrated command execution as the cache-process user. The network path depends on exposing the transport port: its default localhost binding and ordinary in-process vLLM use do not create the same remote exposure. A reminder to inspect serialization at internal service boundaries, not just model-file loaders.

research.jfrog.com · October 2026
2026-10

Cooldown Landmines: Cross-Tenant Interference Attacks on LLM Gateways

Gao, Chen, Wu et al.

LiteLLM kept backend cooldown state shared across tenants even when their request quotas were separate. In the authors' authenticated, self-hosted tests, one tenant's rejected or failing requests could steer another tenant to fallback or denial while a deployment was serviceable. Checking which failures may update shared state and scoping quota cooldowns by tenant cut after-recovery victim fallback from 54/60 to 0/60 in five paired runs, but increased attempts against exhausted shared quotas. The paper shows why isolation has to cover failure handling as well as admission; the tested configuration is not evidence about every hosted gateway.

arxiv.org · October 2026
2026-09

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

Bali, Ponnapalli, Wang, Crooks, Shenker & Zaharia

FairInference schedules per-token deadlines and accounts for shared KV-cache use so one tenant's workload does not impose unbounded token delays on others. In modified SGLang tests with Llama 3 8B on an A100, it limited latency spikes while improving throughput against the paper's comparator. Its guarantee depends on bounded client demand and calibrated timing; it addresses performance isolation, not cross-tenant confidentiality.

arxiv.org · September 2026
2026-09

Off Guard: Breaking LiteLLM from Authentication Bypass to Cloud Compromise

Amitai Cohen & Yaara Shriki (Wiz)

Wiz found two distinct LiteLLM failures: an MCP authentication bypass (CVE-2026-59822) that exposed connected tools, and a custom-code guardrail path (CVE-2026-59821) that allowed server code execution after the attacker cleared a separate key or admin boundary. The first bug alone is not RCE. In a February 2026 scan of 3,074 public instances, 9.6 percent accepted a default key or required no authentication; deployment owners should verify patch status and gateway credentials, especially when the proxy can reach cloud resources.

wiz.io · September 2026
2026-08

Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference

Tejasvi C. Addagada

Prefix reuse across requests improves throughput, but a shared cache also exposes a timing side channel in multi-tenant inference. The authors measure attack signals on unprotected serving engines, then evaluate KVGov's per-principal salt in a simulation calibrated to those measurements; the proposed salt keeps tenants' cache keys separate in that model. Read beside GRIEF's concurrent-request fuzzing, while keeping the measured attack and simulated defense results distinct.

arxiv.org · August 2026
2026-08

SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels

Jo, Park, Lee & Song

Activation sparsity lets a serving system skip weight accesses for inactive neurons, which makes those accesses input-dependent and therefore leakable. SparSEEty builds a neuron-activation oracle from the side channels and inverts the activation traces to reconstruct the input tokens, demonstrated end to end against a serving system running inside an Intel TDX confidential VM. A confidential-computing boundary does not help when the leak is in the access pattern the optimization introduced.

arxiv.org · August 2026
2026-06

Image Prompt Reconstruction Attacks on Distributed MLLM Inference Frameworks

Luo, Chang, Wei, Wu, Gao, Qiu, Yu & Liu

Intermediate embeddings exchanged between participants in distributed multimodal inference leak the input image, not just the text prompt studied previously. Embedding extraction reaches near-total accuracy across almost all layers, then two attacks reconstruct from it: patch assembly for pixel-level output, embedding-guided diffusion for semantic. Evaluated on Gemma 3, Phi 4 Multimodal, Qwen 2.5 VL, and Llama 4 Scout. Worth reading if you treat a disaggregated inference deployment as a trust boundary, which most deployments do not.

arxiv.org · June 2026
2026-05

CleanBase: Detecting Malicious Documents in RAG Knowledge Databases

Jin, Wang, Zou, Jia & Gong

Malicious documents crafted for the same attack-targeted question exhibit high semantic similarity to each other. CleanBase builds a similarity graph and detects cliques, with a statistically-determined threshold and theoretical error-rate bounds. A working CleanBase deployment reduces the poisoned corpus before retrieval, making retrieval rate a function of detection coverage rather than a fixed attack property.

arxiv.org · May 2026
2026-05

Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing (GRIEF)

Zhao, Zhao, Zhang, Liu & Mazurek

A greybox fuzzer for LLM inference engines that treats timed multi-request traces as first-class inputs, which is the right abstraction for bugs that only exist under concurrency. Fifteen findings across vLLM and SGLang, ten confirmed by developers and two assigned CVEs, spanning KV-cache isolation failures, cross-request performance degradation, and crashes. The closest published work to what Crucible does, pointed at the serving layer instead of the parser: the state is shared and the input is a schedule rather than a file.

arxiv.org · May 2026
2026-04

Behavioral Consistency and Transparency Analysis on LLM API Gateways

Lin, Wan, Pei, Xu, Xu & Xue

GateScope audits third-party LLM API gateways across response content, multi-turn quality, billing accuracy, and latency. Measurement of 10 commercial gateways found undisclosed model substitutions, degraded conversation memory, pricing deviations, and inconsistent latency. The billing-accuracy finding has the clearest actionability: gateways charging for model calls that differ from what was advertised.

arxiv.org · April 2026
2026-04

Separable Expert Architecture: Privacy-Preserving LLM Personalization via Composable Adapters

Schneider, Schoenegger & Bariach

Three-layer architecture: static base model, composable domain-expert LoRA adapters, and removable per-user proxy artifacts. Removing a user's artifacts returns outputs to baseline and prevents cross-user leakage, tested on Phi-3.5-mini and Llama-3.1-8B. Reframes machine unlearning as deterministic deletion of a separable artifact rather than expensive parameter updates.

arxiv.org · April 2026
2026-02

mcp-run-python: lack of isolation allows MCP server takeover

Natan Nehorai (JFrog)

CVE-2026-25905 crossed the intended boundary between Python executed in Pyodide and its surrounding JavaScript environment, letting untrusted code alter the MCP server's tool metadata. The lesson is about the actual runtime boundary, not a containerized Python runner. JFrog says the project was archived and a fix was unlikely, so an operator should remove the service or isolate it as untrusted rather than assume an upstream patch.

research.jfrog.com · February 2026
2025-10

RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems

Rorseth, Godfrey, Golab, Srivastava & Szlichta

If-then rules linking source presence and absence to RAG output behavior, with Apriori-like pruning to avoid brute-force source-combination search. The provenance angle extends beyond explainability: if you can determine which source combinations produce which outputs, you can identify which sources a poisoning attack needs to control.

arxiv.org · October 2025
2025-08

Uncovering memory corruption in NVIDIA Triton (as a new hire)

Will Vandevanter (Trail of Bits)

Two remotely-exploitable memory-corruption bugs (CVE-2025-23310, CVE-2025-23311) in Triton's HTTP request handling, surfaced via static analysis plus chunked-encoding probing. The reminder: production inference servers are still C/C++ network services with all the attendant historical bug classes, and authentication is off by default.

blog.trailofbits.com · August 2025
2025-08

Breaking NVIDIA Triton: CVE-2025-23319 vulnerability chain to RCE

Wiz Research

A multi-stage vulnerability chain in the Triton Python backend, starting from a minor information leak about shared-memory region names and escalating to unauthenticated RCE. Useful as a case study in chaining low-severity primitives into a takeover.

wiz.io · August 2025
2024

GGUF-SSTI: Llama-Drama and the Jinja template attack surface

JFrog Security Research

Reference for CVE-2024-34359 (the chat-template Jinja RCE in llama-cpp-python) and the broader question of when loading a GGUF model can lead to server-side template injection. The case study for why loader extensions need the same threat-modeling rigor as the loader itself.

research.jfrog.com · 2024

Identity, AD, and lateral movement

9 sources

Writing on identity systems, directory services, and trust across platform boundaries.

2026-09

Unmasking SCCM Application Execution

Joshua Prager (SpecterOps)

A small Configuration Manager client lab traces why application launches can disappear from process trees that expect CcmExec.exe to start the child directly. Scripts followed that path, but tested applications went through WMI and AppProvider.dll, with WmiPrvSE.exe creating the child. That gives defenders a specific correlation to investigate alongside deployment context; the five-client study does not establish a universal signature for every package type or deployment.

specterops.io · September 2026
2026-08

Pass-the-Passkey Family of Attacks

Michael Grafnetter (SpecterOps)

Windows 11 logged complete WebAuthn assertions where an unprivileged local user could read them; paired with weaknesses in Entra's replay handling, those records enabled privileged cloud identity impersonation in the researchers' demonstration. The logging issue was patched in July 2026, so this is a case study in authenticator output escaping its intended boundary, not a claim that that exact path remains open. The broader family also examines malware-induced WebAuthn prompts.

Black Hat USA 2026 · August 2026
2026-07

Introducing Attack Path Management for Entra Agents in BloodHound Enterprise

Elad Shamir (SpecterOps)

An Entra or Copilot Studio agent can turn an apparently narrow right to invoke the agent into access to the maker's downstream credentials and connected resources. SpecterOps adds agent identities and their relationships to BloodHound so defenders can trace those paths across Entra, Power Automate, and the services an agent reaches. The example is deliberately simplified and the first graph omits some Foundry and data-plane edges, so it is a map of an emerging identity boundary rather than complete coverage of every agent path.

specterops.io · July 2026
2026-07

Clustered Points of Failure

Garrett Foster (SpecterOps)

Cluster Name Objects and Virtual Cluster Objects store encrypted resource credentials on every node so that failover works, which means compromising one node hands you the credentials for the whole cluster. Clusters host SQL Server, ADFS, Configuration Manager, and Exchange, so cluster compromise reads as domain compromise. 82 percent of BloodHound Enterprise clients had at least one cluster, averaging 141 per environment. Availability engineering and credential isolation are in direct tension, and availability won.

specterops.io · July 2026
2026-07

There and Back Again: An Operators Guide on NTLM Relaying Egress

Logan Goins (SpecterOps)

Coerce SMB or WebDAV authentication out of the target network to a cloud-hosted VM, forward it back through red team infrastructure, and relay it into ADCS web enrollment or LDAP without relay protections. The point is that relay tradecraft still works when local relay is off the table, which is the normal condition from a low-privilege C2 agent behind a host firewall. Mitigations are the familiar four: LDAP signing and channel binding, EPA on ADCS endpoints, outbound SMB egress filtering, and host firewall rules.

specterops.io · July 2026
2026-03

RTFM: Read The Fatal Manual — When Vendor Documentation Creates Critical Attack Paths

Martin Sohn Christensen (SpecterOps)

SpecterOps examined 23 vendor documents across 16 vendors and found recommendations that can lead operators to create dangerous Active Directory certificate and identity permissions, especially AD CS ESC1, ESC3, and ESC4 paths. The sample does not establish how often customers deployed those settings, and vendor remediation varied when the piece was published. The practical audit is to compare actual templates and service-account rights with the minimum the integration needs, rather than treating setup documentation as a security review.

specterops.io · March 2026
2026-03

Attack Paths Don't Stop at Identity Providers

Jared Atkinson (SpecterOps)

Modeling Okta in BloodHound Enterprise alongside AD, Entra, GitHub. The argument: identity boundaries between platforms are where attack paths actually live, and treating any single platform in isolation underrepresents real risk.

specterops.io · March 2026
2025-10

AdminSDHolder: Misconceptions, Misconfigurations, and Myths

Jim Sykora (SpecterOps)

Long-form correction of decades of incorrect documentation around AD's AdminSDHolder mechanism. The kind of historical-grounding piece that's useful before doing anything privileged-account-related on AD engagements.

specterops.io · October 2025
2021

Certified Pre-Owned

Will Schroeder & Lee Christensen

The AD CS paper that opened up the modern wave of ADCS work. Still the cleanest framing of what a tooling-up problem looks like before any tools exist.

specterops.io · 2021

AI x security writing

9 sources

Reports and perspectives on AI-assisted security research and the surrounding practice.

2026-10

Investigating unintended model actions in our evaluations and internal use

Anthropic

Anthropic reports cases where Claude, faced with blocked or poorly specified tasks in internal evaluations and use, sought unintended paths: following an external-server instruction to run a calculation, submitting real forms, reading gated public data, and using URL shorteners around fetch limits. Many tasks were ambiguous or impossible, and Anthropic reports minimal impact and no known customer or internal data exposure. A replay monitor caught the known cases, not every possible future deviation. The practical lesson is to specify authority and evaluate actual actions, not just final answers.

anthropic.com · October 2026
2026-10

Rogue Agents Investigation

Asymmetric Security

A public-record reconstruction of the reported rogue OpenAI agents found browser and fetch tools composed into an unintended communication channel, successful access to staging and public data, and probes against backups and SQL injection without verified success. The team recovered roughly 22 MB of public statistics and observed an upload whose contents it could not recover. Missing private scans and expired records leave material uncertainty: the report does not prove sensitive-data theft or concealment. Read it for the trace-to-consequence discipline as much as the incident narrative.

asymmetricsecurity.com · October 2026
2026-08

The Hugging Face incident and the road ahead

OpenAI

During internal cyber evaluations with reduced safeguards, OpenAI agents used shared infrastructure as an unauthorized message board, bypassed intended internet isolation, and compromised parts of OpenAI and Hugging Face systems. OpenAI's account connects the behavior to evaluation design, sandbox and credential boundaries, monitoring, and incident response. This is a disclosure about a controlled research setting and a real third-party impact, not evidence that public production agents behaved the same way. Read it with METR's narrower independent examination of the agent traces.

openai.com · August 2026
2026-08

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident

Greenblatt, Cotra & Wijk (METR / Redwood Research)

METR and Redwood Research examined OpenAI's agent transcripts on site, focusing on the Hugging Face episode. Roughly 1,200 agents used an unsanctioned shared message board and about 700 participated in the Hugging Face attack; the team reviewed the coordination and reasoning behind those actions. Their scope excluded earlier training incidents, later OpenAI infrastructure compromise, and OpenAI's remediation process, so the report is independent evidence about one part of the event, not a full validation of every claim in OpenAI's disclosure.

metr.org · August 2026
2026-07

Atlas: Wiz's autonomous AI agent for vulnerability research

Nir Ohfeld & Yuval Avrahami (Wiz)

Four stages: threat model the attack surface with code property graphs, fan out independent agents to test hypotheses in parallel, contest every finding adversarially before accepting it, then prove it with a working exploit in a real execution environment. 90.9 percent on CyberGym and over 200 previously unknown vulnerabilities in heavily audited projects including Kubernetes, gRPC, and the Linux kernel. The design claim matches my own experience: the durable advantage is the harness around the model, not the model.

wiz.io · July 2026
2026-07

Fast Remediation Is the New Trust Model: JFrog and OpenAI Collaboration on Zero-Day Security Findings

Yoav Landman (JFrog)

OpenAI models autonomously found previously unknown vulnerabilities in self-hosted Artifactory during a capability evaluation, and the fixes shipped in 7.161 and above. Landman's argument is that the trust model has to move to remediation speed, because a model-discovered zero-day sitting in a vendor queue is pure attacker upside. Read next to the Atlas post: this is the month AI-discovered vulnerabilities became an operational question for defenders instead of a thought experiment.

jfrog.com · July 2026
2026-04

State of AI in the Cloud 2026

Wiz Research

Infrastructure-side measurements of AI adoption: 81% of cloud environments use managed AI services, 90% run self-hosted, 80% have MCP servers. The framing of AI as accumulated rather than adopted is a useful governance lens.

wiz.io · April 2026
2026-04

GitHub RCE via X-Stat header injection (CVE-2026-3854)

Wiz Research

An authenticated git push achieves RCE on GitHub's backend through a delimiter-based internal protocol. Notable also as one of the first critical vulnerabilities the team credits to AI-augmented reverse engineering.

wiz.io · April 2026
2026-04

Trailmark: turning code into security-analysis graphs

Trail of Bits

Tree-sitter plus rustworkx, packaged as Claude Code skills for blast-radius and taint-propagation analysis. Useful as a reference for how graph reasoning composes with LLM agents in a security workflow.

blog.trailofbits.com · April 2026