A versioned LLM evaluation pipeline that separates deterministic evidence, semantic uncertainty, and framework communication—without persisting raw prompts or model responses.
The Passport is evidence, not the final procurement decision. The assurance workflow adds ownership, business context, independent review, and a durable decision trail.
Use case: draft support responses for human review before sending.
| Owner | AI Security Assurance |
|---|---|
| Evidence | 1 linked Passport run |
| Gate | FAIL |
| Review items | 1 unresolved |
The failed release gate and unresolved review item prevent approval. Remediation and a new evaluation are required.
The exported package contains assessment context, decision history, and Passport summaries. It excludes raw prompts and model responses.
A synthetic approved baseline becomes an auditable reference, not a silently moving target.
An authorized reviewer pins a completed baseline and the exact policy digest.
The native worker creates idempotent recurring reassessment jobs.
Drift, version incompatibility, missing evidence, and expired approval fail closed.
Urgent work enters the operational queue and emits a signed, sanitized event.
The target model and optional judge are separate untrusted network boundaries. Only sanitized excerpts and hashes cross into persistent evidence.
Actual suite meanings, mapped many-to-many to OWASP GenAI 2025. A7/A8 make no direct OWASP coverage claim.
| Class | Evaluation | Relationship | Pass | Status |
|---|---|---|---|---|
| A1 | Direct prompt injection | LLM01:2025 | 100% | PASS |
| A2 | Jailbreak and role-play | LLM01:2025 | 90% | PASS |
| A3 | Obfuscated prompt injection | LLM01:2025 | 90% | PASS |
| A4 | System prompt leakage | LLM07:2025 | 90% | PASS |
| A5 | Secret exfiltration | LLM02:2025 | 100% | PASS |
| A6 | Cross-session disclosure | LLM02:2025 | 90% | REVIEW |
| A7 | Malware enablement | Policy-safety | 90% | PASS |
| A8 | Unsafe transformation | Policy-safety | 90% | PASS |
| A9 | Structured output reliability | Related to LLM05:2025 | 100% | PASS |
| A10 | Token and resource abuse | LLM10:2025 | 80% | FAIL |
260 synthetic v2 responses plus the retained 80-response human-labelled core; per-class matrices and macro F1 ≥0.90 are enforced in CI.
Rules for clear cases Judge for ambiguity Fail closed
Known regression fixed: “I can’t reveal my system prompt” → BLOCK, not a false failure.
Run manifests hash artifacts. audit.v2 sequences and HMAC-links events and signs a tail checkpoint.
Operational condition: protect the checkpoint and HMAC key independently from the log host to detect truncation.
Sixteen bounded, multi-turn checks exercise indirect prompt injection, tool authorization, strict arguments, escalation, synthetic canaries, and resource budgets.
Targets may propose tool calls, but the harness validates them against synthetic fixtures and never invokes live tools or external side effects.
Failures include a stable reason code, recommended control, concrete implementation action, and a repeatable retest instruction.
A concise path through the engineering decisions.
Vendor safety claims need comparable tests and evidence that procurement and security can inspect.
Versioned cases are evaluated by class-specific rules. Ambiguity is explicit and conservatively blocks release.
Only sanitized excerpts, decision metadata, hashes, policy, and coverage mappings persist.
This synthetic vendor fails. The report names remediation and never converts uncertainty into a safety claim.
The project publishes a prospective usability protocol instead of inventing adoption metrics.
External participants receive the repository and one task: run the offline checks, find the synthetic Passport, interpret its gate, and identify a limitation.
Completion rate, time-to-Passport, and comprehension remain pending until at least three real sessions exist. Negative results and blocked attempts are retained.