AI Bill of Materials
Acme Resolver
Version 2.3.1 · Customer support triage agent
UID
urn:aibom:acme-resolver:prod
Schema
CycloneDX-AI 1.6 (draft)
Status
Production
EU AI Act
Limited risk
§01System Identity
Owner
Customer Experience Engineering — Maya Okafor
Regions
EU-West · US-East · US-West
Purpose / intended use
Ingests inbound customer support tickets, classifies urgency and topic, drafts an initial response, and routes complex or sensitive cases to a human agent queue with a recommended action.
Out-of-scope uses
Not used for hiring, credit, insurance, healthcare diagnosis, or any decision with legal or significant individual impact. Not used to auto-send replies without human review for refund, billing, or compliance topics.
§02AI Components
| Name | Provider | Model ID | Role | Hosting | Pinning |
|---|---|---|---|---|---|
| Primary inference model | Anthropic | claude-sonnet-4-20250514 | Primary inference (classification + draft reply) | vendor-hosted (US (Anthropic, ZDR enabled)) | Pinned |
| Embeddings model | OpenAI | text-embedding-3-small | Embeddings (semantic search over knowledge base) | vendor-hosted (US (OpenAI)) | Pinned |
§03Training Data & Lineage
First-party training or fine-tuning
No — relies on provider disclosures
Provider transparency disclosures
- https://www.anthropic.com/claude/model-card
- https://platform.openai.com/docs/models/embeddings
Training cutoff notes
Claude Sonnet 4 training cutoff: early 2025. Acceptable for support workflows; agent does not produce time-sensitive factual claims about post-cutoff events without retrieval grounding.
Retrieval-augmented generation
Knowledge base
Acme product documentation, public help center articles, internal support playbooks, and recent resolved-ticket exemplars (anonymized). ~14,000 chunks.
Indexing approach
Documents chunked at ~500 tokens with 50-token overlap, embedded with text-embedding-3-small, indexed in Pinecone (us-east-1). Top-8 retrieval with MMR re-ranking.
Refresh cadence
Documentation: nightly. Internal playbooks: on commit. Resolved-ticket exemplars: weekly batch with PII scrub.
Source provenance
Every retrieved chunk carries source URL or internal doc ID; sources are surfaced in the agent draft for human reviewer verification.
§04Data Flows
| Source | Destination | Data types | Sensitivity | Encryption | Retention | Cross-border |
|---|---|---|---|---|---|---|
| Customer ticket (web form / email) | Model context (Claude Sonnet 4) | Free text input, Customer data, Occasional PII | Confidential | TLS 1.3 | 30 days in queue, then archived 12 months for audit | EU-customer tickets routed via US Anthropic endpoint (ZDR enabled, no training) |
| Model output (classification + draft reply) | Human agent queue (Acme Helpdesk) | Customer data, AI-generated text | Confidential | TLS 1.3 | Linked to ticket; deleted with ticket per retention policy | No |
| Pinecone vector index | Model context | Internal documentation, Public-only content | Internal | TLS 1.3 | Index persistent; re-built nightly | No |
§05Third-Party Dependencies
| Category | Name | Version | Notes |
|---|---|---|---|
| AI provider | Anthropic Messages API | — | ZDR (Zero Data Retention) enabled per contract. |
| AI provider | OpenAI Embeddings API | — | |
| Vector DB | Pinecone | Serverless, us-east-1 | |
| Orchestration | Vercel AI SDK | 5.0.x | |
| Observability | Datadog LLM Observability | — | |
| Moderation | OpenAI Moderation endpoint | — | Pre-filters inbound tickets for policy violations. |
§06Human Oversight & Controls
HITL checkpoints
Every AI-drafted reply is reviewed by a human support agent before being sent. The agent UI presents the draft alongside cited retrieved sources and the classification rationale. Agents accept, edit, or discard.
Approval-required actions
Refund or credit decisions · Account suspension or closure · Any reply to legal, regulatory, or media-flagged inquiries
Escalation paths
Low-confidence classifications (model confidence < 0.6) or detected sensitive topics (legal, medical, abuse) skip the auto-draft step and route directly to a senior agent.
Override capabilities
Agents can override classification and routing at any point. Override events are logged and feed weekly model-performance reviews.
User-facing AI disclosure
Yes — Help center page and ticket-confirmation email state: 'Acme uses AI to help our team respond faster. A human reviews every reply before it reaches you.'
Right to human review
Yes — Customer can reply 'speak to a human' or click an opt-out link in the ticket confirmation to bypass AI-assisted triage entirely.
§07Risk Classification
EU AI Act tier
Limited
NIST AI RMF
Medium — narrow purpose, human-in-the-loop, no autonomous high-impact decisions.
Internal rating
Medium (per Acme AI Governance Policy v3)
Documented concerns
Documented misclassification of urgency for non-English tickets, particularly Spanish and Portuguese. Documented mild bias toward formal language in draft replies (less effective for casual or distressed phrasing).
§08Evaluation & Monitoring
Pre-deployment evaluation
Held-out set of 2,400 historical tickets across topic and urgency labels. Classification F1 = 0.87 on English; 0.71 on non-English subset (gap flagged in launch review).
Bias / fairness testing
Drafts evaluated for tone parity across customer name proxies and stated regions. No significant disparity in draft length or sentiment detected; formality bias acknowledged as a known limitation.
Red-team / adversarial testing
Internal red-team exercise covered prompt injection via ticket content, attempted refund-policy bypass, and PII exfiltration via knowledge-base poisoning. No critical findings; two medium findings remediated pre-launch.
Production monitoring
Datadog LLM Observability tracks per-prompt latency, classification confidence distribution, agent override rate, and hallucination flag rate (sources cited vs. claims made). Drift alerts on confidence distribution shift > 10%.
Logging & audit
All prompts, completions, retrieved sources, classification scores, agent overrides, and final sent replies are logged with ticket ID and retained 12 months. Audit trail accessible to Security and Compliance.
Incident history
One incident (2025-Q4): RAG returned stale refund-policy chunk for a 36-hour window after a policy change; caught by agent review, no incorrect replies sent. Root cause: nightly index re-build missed urgent policy update. Remediation: on-demand index invalidation hook for policy-tagged documents.
§09Lifecycle & Governance
Deployed at
2025-11-04
Review cadence
Quarterly governance review; monthly performance review; weekly ops review
Sunset criteria
Decommission if agent override rate exceeds 35% for two consecutive months, if a Severity-1 incident occurs, or if Anthropic deprecates the pinned model without a validated successor.
Change management
Prompt and configuration changes go through a PR review with Compliance sign-off for material changes. Model version updates are pinned and require re-running the evaluation suite before promotion.
Approval workflow
Engineering owner + Compliance reviewer + Security reviewer for any material change to prompts, model version, data flows, or retention.
Governance owner
Acme AI Governance Committee (Chair: VP Engineering)
§10Known Limitations & Disclosed Risks
Failure modes
Misclassification of urgency for non-English tickets; occasional over-confident draft replies for ambiguous billing questions; retrieval can miss very recent documentation if index is mid-refresh.
Off-label uses
Do not use the underlying components for hiring, credit, insurance, healthcare, or any individual decision with significant impact. Do not enable auto-send without human review.
Performance boundaries
Works well for: account questions, product how-to, basic billing, password reset, shipping status. Works less well for: complex billing disputes, multi-product accounts, non-English tickets, emotional or crisis-tone messages.
Disclosed biases
Mild formality bias in draft phrasing. Documented language-coverage gap.
Caveats
Confidence scores are not calibrated probabilities — treat as relative signal only. Cited sources reflect retrieval, not verification — agents must verify before sending.