# AgentProv methodology

Canonical page: https://agentprov.com/methodology

Every number on agentprov.com comes from the public incident catalog (https://github.com/webpro255/awesome-ai-agent-attacks), parsed at build time, and from layer tags assigned by hand under the rules below. 385 of 385 incidents are tagged: 270 in-model and 115 outside the model.

## What the catalog includes

Each entry names a specific affected company, product, or project and records a date, a concrete impact, a technical root cause, any CVE, and at least one verifiable source from a reputable publication or official advisory.

The catalog accepts:

- documented AI agent security incidents with real-world impact
- agent framework and tool vulnerabilities with published advisories or CVEs
- AI chatbot integration breaches affecting real organizations
- supply chain attacks targeting AI development tools and libraries
- research disclosures demonstrating exploitable vulnerabilities in production AI systems
- autonomous agent behavior incidents, such as unintended actions and sandbox escapes

It excludes hypothetical attacks, unverified claims, commentary, vendor marketing, and duplicates. Inclusion in the catalog does not mean an incident is in the model: the tagging rules decide that separately, and entries outside the model are counted, not dropped.

## Tagging rules

Rules for `data/tags.yaml`. The eight layers are defined on the Model page; this file records how incidents are assigned to
them. `npm run data` enforces the schema.

### Schema

In-model row:

```yaml
<incident-slug>:
  primary: 4              # exactly one layer number, 1..8
  secondary: [1]          # zero or more, never the primary
  actor: attacker         # optional: vendor_eval | operator | attacker | unknown
  related: [<slug>]       # optional, any row: the same event described from another side
  fixable: true           # true, false, or null
  vendor_response: patched
  in_the_wild: true
```

Outside-the-model row (no primary or secondary):

```yaml
<incident-slug>:
  out_of_model: true
  class: capability       # capability | offensive_use | resource_theft | context | model_safety | adjacent | reliability
  fixable: true
  vendor_response: patched
  in_the_wild: false
```

Each row gets a one-line `#` comment above it with the reason, citing the Root Cause or Impact text.

### In or out of the model

An incident is in-model only when agent tooling was the attack surface: the agent, its tools, runtime,
config, supply chain, or credentials. Otherwise it is `out_of_model: true` with one class:

- `capability`: an AI found or built an exploit against non-agent software.
- `offensive_use`: AI used as the instrument against humans or non-agent systems, including impersonating
  an AI product as a lure.
- `resource_theft`: the key was stolen or leaked outside agent tooling (infostealer, phishing, code leak,
  credential resale) and then spent on inference or agent compute (LLMjacking).
- `context`: regulatory, legal, and policy events, trend reports, and surveys that are not themselves an
  incident. Kept because the report cites them, so the catalog stays fully tagged.
- `model_safety`: jailbreaks and harmful-content generation where the requester is the user, nothing
  acts on a system, and no third party is harmed. A guardrail is a content judgment, not a provenance
  control; defeating one with no action or target is out of the model.
- `adjacent`: ecosystem supply chain compromises with no agent or AI involvement as surface,
  instrument, capability, or payload target.
- `reliability`: an AI system producing harmful or false output, or failing at its task, with no
  attacker, no action on a system, and no security angle.

The class list is frozen at seven: capability, offensive_use, resource_theft, context, model_safety,
adjacent, reliability. A future row that fits none of them is a catalog scope question, not a new class.

Worms and compromised packages (the Shai-Hulud lineage and similar): in-model, primary by the supply
chain rule and secondary 1, when the payload harvested agent or AI tooling credentials and configs, or
the compromise spread through agent tooling. `adjacent` when the AI angle is only that an AI-adjacent
vendor was hit.

Agent runtime includes inference servers, LLM gateways, memory stores, vector databases, model
registries, and ML platforms. Incidents against them are in-model.

Layer 1 primary covers any case where the run held a credential it should not have: over-broad,
longer-lived than the run, or mishandled by the platform (stored unencrypted, exposed by an endpoint,
leaked by the tool, or not bound to its issuer). `resource_theft` stays for keys stolen outside agent
tooling and then spent.

An agent that harms a third party with no attacker involved is in-model; the failed link is whichever
control should have stopped the action.

Model Capability Extraction: when the attacker defeated a platform control (for example, tricking the
model into decrypting its own protected output), it is in-model at the layer of the defeated control.
When extraction used ordinary API access at scale with no control defeated (distillation campaigns), it
is `out_of_model`, class `context`.

Trusted AI Platform Abuse: in-model layer 4 when agent tooling executed or loaded the hosted artifact (a
malicious model file loaded by a tool, a package pulled by an agent). `out_of_model`, class
`offensive_use`, when the hosted artifact was a lure aimed at humans.

Fabricated Security Intelligence: in-model layer 7 when a review or approval process consumed the
fabricated evidence (a maintainer acted on it, a CVE or advisory was issued, a fix was merged).
`out_of_model`, class `offensive_use`, when it was only submitted or published and nobody acted.

Outside-the-model rows are counted and charted on /stats ("Outside the model"), never dropped.

### Choosing the primary layer

- The primary is the link whose failure made the incident possible, not where the damage showed up.
- When unsure between two layers, pick the earlier number and say so in the reason. A taxonomy mapping
  (for example, Agent Sandbox Escape is layer 6) beats this tiebreak.
- An exposed server or endpoint with no authentication is primary 6. When it executes code, primary 6
  with secondary 5; code execution that needs an authenticated user, or that the model emits, is
  primary 5.
- A network path out of a sandbox (DNS, proxy, open egress) is an isolation failure, primary 6. Layer 8
  is primary only when the harm is data sent to a destination the agent chose (exfiltration,
  publication).
- An SSRF primitive is primary 8, with 6 secondary when it is reachable without authentication.
- A config or instruction file that runs or loads on open or clone is primary 5.
- An injected page or document that steers the agent is primary 2; add 8 if data left through an
  agent-chosen destination, and 1 if credentials were taken.
- Supply chain:
  - primary 4 when the artifact's origin was the first failure (typosquat, hallucinated name, maintainer
    account socially engineered);
  - primary 1 when a leaked or stolen credential is the documented root cause, or when an account was
    compromised by an unknown method;
  - primary 7 when the publish pipeline accepted what it should not have (forged provenance, an
    unreviewed push feeding a release).
- Oversight integrity is layer 7: monitoring disabled, tests altered to hide it, approval prompts that
  misdescribe an action.

### Field meanings

- `actor` (optional, in-model rows only): who drove the failing action.
  - `vendor_eval`: the vendor's own evaluation or research agent caused the harm (third-party evaluators
    running agents count here too).
  - `operator`: a deployed agent acting for its operator, with no attacker involved.
  - `attacker`: an attacker drove the agent or exploited the platform. Vulnerability disclosures use
    this, since exploitation needs an attacker.
  - `unknown`: not determinable from the entry.
- `related` (optional, any row): slugs of other incidents that describe the same event from another
  side. Each row keeps its own tags; the validator checks the slugs exist.

- `fixable`: whether a code fix closes the class. true (a vendor patch closed it), false (architectural),
  null (unknown).
- `vendor_response`: what the vendor did, independent of `fixable`. An architectural incident can still be
  `patched` when the vendor changed its process or product. `patched | disputed | not_a_vulnerability |
  no_response | mixed | unknown`. Use `mixed` for multi-vendor advisories where vendors responded
  differently, and give the per-vendor split in the reason. A researcher saying there is nothing to patch
  is not a vendor response.
- `in_the_wild`: true if real victims or active exploitation of this incident's failure were reported.
  Internal red-teaming is false, even when a byproduct CVE is later exploited elsewhere.

## Outside the model, by class

- AI capability: 20
- Offensive use: 33
- Resource theft: 5
- Context: 31
- Model safety: 3
- Adjacent ecosystem: 21
- Reliability: 2

## Downloads

- https://agentprov.com/data/incidents.csv: The parser output: one row per catalog incident.
- https://agentprov.com/data/incidents.json: One object per incident with layer tags merged in.
- https://agentprov.com/data/tags.yaml: The hand-assigned layer tags, with a one-line reason per row.
- https://agentprov.com/data/stats.json: The aggregates behind every chart on the site.

## Catalog scope review

Entries tagged adjacent: ecosystem compromises with no agent or AI involvement, listed for a review of whether they belong in the catalog.

- 2026-07-06: [Summer.fi "Keeper AI Agents" Exploit Drains About $6M](https://agentprov.com/incidents/2026-07-06-summer-fi-keeper-ai-agents-exploit-drains-about-6m)
- 2026-06-17: [Mastra AI npm Scope Compromise (Sapphire Sleet / UNC1069)](https://agentprov.com/incidents/2026-06-17-mastra-ai-npm-scope-compromise-sapphire-sleet-unc1069)
- 2026-06-03: [node-gyp "Phantom Gyp" Self-Propagating npm Worm (Miasma)](https://agentprov.com/incidents/2026-06-03-node-gyp-phantom-gyp-self-propagating-npm-worm-miasma)
- 2026-06-01: [Red Hat @redhat-cloud-services npm "Miasma" Worm](https://agentprov.com/incidents/2026-06-01-red-hat-redhat-cloud-services-npm-miasma-worm)
- 2026-05-28: [Nx Console Malicious VS Code Extension Leads to GitHub Repository Breach](https://agentprov.com/incidents/2026-05-28-nx-console-malicious-vs-code-extension-leads-to-github-repository-breach)
- 2026-05-25: ["Megalodon" Mass GitHub Actions Secret Exfiltration](https://agentprov.com/incidents/2026-05-25-megalodon-mass-github-actions-secret-exfiltration)
- 2026-05-18: [actions-cool GitHub Actions Tag Hijack (Mini Shai-Hulud)](https://agentprov.com/incidents/2026-05-18-actions-cool-github-actions-tag-hijack-mini-shai-hulud)
- 2026-05-14: [OpenAI Internal Source Code and Certificate Theft via TanStack Worm](https://agentprov.com/incidents/2026-05-14-openai-internal-source-code-and-certificate-theft-via-tanstack-worm)
- 2026-04-30: [PyTorch Lightning PyPI Compromise (Mini Shai-Hulud)](https://agentprov.com/incidents/2026-04-30-pytorch-lightning-pypi-compromise-mini-shai-hulud)
- 2026-04-22: [Xinference PyPI Package Compromise (Versions 2.6.0-2.6.2)](https://agentprov.com/incidents/2026-04-22-xinference-pypi-package-compromise-versions-2-6-0-2-6-2)
- 2026-04-08: [UNC1069 Contagious Interview Cross-Ecosystem Package Campaign](https://agentprov.com/incidents/2026-04-08-unc1069-contagious-interview-cross-ecosystem-package-campaign)
- 2026-04-01: [Drift Protocol $285M Exploit](https://agentprov.com/incidents/2026-04-01-drift-protocol-285m-exploit)
- 2026-03-31: [Axios npm Supply Chain Attack](https://agentprov.com/incidents/2026-03-31-axios-npm-supply-chain-attack)
- 2026-03-31: [Cisco Source Code Stolen via Trivy Breach](https://agentprov.com/incidents/2026-03-31-cisco-source-code-stolen-via-trivy-breach)
- 2026-03-23: [Checkmarx KICS GitHub Actions Compromise](https://agentprov.com/incidents/2026-03-23-checkmarx-kics-github-actions-compromise)
- 2026-03-20: [CanisterWorm npm Worm by TeamPCP](https://agentprov.com/incidents/2026-03-20-canisterworm-npm-worm-by-teampcp)
- 2026-03-19: [Trivy GitHub Action Compromise by TeamPCP](https://agentprov.com/incidents/2026-03-19-trivy-github-action-compromise-by-teampcp)
- 2026-03-11: [UNC6426 nx npm to AWS Admin Takeover](https://agentprov.com/incidents/2026-03-11-unc6426-nx-npm-to-aws-admin-takeover)
- 2025-03-15: [tj-actions/changed-files GitHub Actions Supply Chain Attack](https://agentprov.com/incidents/2025-03-15-tj-actions-changed-files-github-actions-supply-chain-attack)
- 2025-02-21: [Bybit $1.5B Cryptocurrency Heist](https://agentprov.com/incidents/2025-02-21-bybit-1-5b-cryptocurrency-heist)
- 2024-12-04: [Ultralytics PyPI Supply Chain Attack](https://agentprov.com/incidents/2024-12-04-ultralytics-pypi-supply-chain-attack)