Content Moderation System

A/B Testing taught you how to decide whether a model change should ship. This capstone asks a harder question: how do you give each user-generated message, listing, photo, and appeal the policy decision path its product surface requires?

You are the Staff AI Engineer at LogiFlow, the global e-commerce and logistics platform that moves 18 million orders per day across 180 countries. Your sellers post 2.4 million new listings every 24 hours. Buyers and sellers chat live inside the app about shipping windows, returns, and product authenticity. Chat messages need a synchronous send decision; new listings can instead remain pending while slower image or appeal workflows complete. Each surface needs an explicit moderation contract because false negatives harm users while false positives interrupt legitimate commerce.

A real-time content moderation system makes policy decisions fast enough for the surfaces that require synchronous enforcement. This capstone uses a design target of 10K requests per second (RPS) and a sub-200 ms p95 chat decision budget to reason about classifiers, fingerprint lookups, policy-aware judges, restricted review paths, appeals, and versioned policy releases. Those numbers are scenario requirements, not universal product guarantees.

The fulfillment-center analogy is useful. Barcode scanners and weight checks catch predefined problems quickly. Vision systems and rules engines flag suspicious packages for secondary inspection. Ambiguous or high-impact cases reach authorized specialists. Content moderation follows the same pattern: route routine high-confidence traffic through validated fast paths, and reserve contextual model or reviewer work for cases that need it.

Designing this system means balancing the throughput of lightweight classifiers with the context available to slower policy-aware models and reviewers. A small BERT (Bidirectional Encoder Representations from Transformers)-style classifier can score common categories cheaply after it is evaluated on local traffic, while novel exceptions and cross-turn meaning often need richer context. The engineering challenge is to route each content type through an enforcement path whose latency, error costs, and appeal requirements are measured.

System requirements

A production-grade moderation system for this marketplace scenario has operational constraints that must be defined before model selection. It should minimize friction for legitimate users while handling likely violations under a declared policy and review process.

Latency

For this design exercise, target <200 ms p95 (95th percentile) for chat send decisions and <1 s for a listing to receive either a publish decision or an explicit pending-review state. Allocate that budget across gateway, deterministic checks, classifiers, and any synchronous escalation; measure user impact rather than asserting a universal latency threshold.

Throughput

Handle the stated 10K+ RPS target with varying load spikes. E-commerce platforms experience traffic surges during flash sales and holiday shipping windows. Capacity planning needs both steady-state and burst assumptions, plus queue limits for asynchronous review.

Flexibility

Support evolving content policies without requiring a full model retrain for every change. A new counterfeit wording pattern may require a versioned rule or policy-pack update first, followed by regression evaluation, approval, monitored release, and later classifier retuning or retraining.

Multimodality

Moderate text, images, video, and audio simultaneously. Users interact across multiple formats, and malicious actors often hide violations in cross-media contexts (for example, placing hateful text over a benign image). On e-commerce platforms, this includes counterfeit product photos, misleading unboxing videos, and abusive audio in customer support calls.

Due Process

Support transparent appeal workflows and appropriately restricted review loops. Automation errors are inevitable, so affected users need an explainable remediation path; highly sensitive safety reports require a separate authorized process rather than a generic review queue.

Architecture

A useful production pattern is a cascade or tiered architecture. Content flows through a funnel of differently scoped controls: deterministic signatures and rules, learned classifiers, policy-aware review models, and authorized human workflows. Later stages can use more context, but they are not automatically more correct; each action path needs its own evaluation.

By placing validated, lightweight controls at the top of the funnel, the architecture can preserve low latency for routine traffic. Ambiguous, context-dependent, or high-impact content can be held or escalated to slower paths without making every request pay that cost.

Avoid making every request pay for a contextual judge. Let validated Tier 1 paths handle routine decisions, while uncertainty and high-impact actions follow the policy-defined escalation path.

A concrete LogiFlow example: why the cascade matters

Three pieces of seller-generated content arrive in the same second on LogiFlow:

1. Seller listing description: "Bulk packing boxes, pack of 20, ships tomorrow." Deterministic checks find no prohibited signature and a validated low-risk listing classifier stays below its review threshold. The system records an ALLOW under the current listing policy scope. It does not treat a seller identity or familiar wording as proof that every future listing is safe.

2. Buyer-seller chat turn: "I will come to your warehouse tonight and destroy your inventory." A violence classifier scores 0.97. If that score exceeds a validated chat-threat block threshold, the message is held before delivery and routed under the threat-response policy; the decision and policy version are logged.

3. Seller photo + caption: A picture of a handbag with the text overlay "100% authentic Louis Vuitton (limited drop), $89." Image and text signals exceed a review threshold but not an auto-block threshold. The listing remains pending while Tier 2 receives the signals, listing context, and policy clause, then recommends human review. If an appeal later overturns enforcement, that labelled outcome becomes a candidate hard negative for evaluation and retraining.

Without a cascade, every upload pays for contextual inference or enforcement depends on controls too crude for context-dependent cases. A tiered design lets LogiFlow benchmark a low-latency routine path while holding uncertain or high-impact listings for deeper review.

Tiered approach

To balance speed and contextual understanding, the system divides the workload across three distinct operational tiers. Each tier serves a specific purpose in the funnel, filtering out clear-cut cases and escalating only what it can't confidently resolve.

Tier 1: Fast classifier (synchronous path)

The first line of defense is a high-performance, specialized model (typically a distilled BERT, DeBERTa (Decoding-enhanced BERT with disentangled attention), or even a fastText (a library for efficient learning of word representations and text classification) classifier). These are smaller, faster versions of large transformers, trained on labeled data to detect specific categories of violations.

Characteristics

• Architecture: DistilBERT, MobileBERT, or DeBERTa-v3-xsmall.
• Task: Multi-label classification (Output: [prob_hate, prob_violence, prob_spam, ...]).
• Precision: Tuned for high precision on "Block" (don't auto-block unless sure) and high precision on "Allow" (don't auto-allow unless sure). Everything else goes to Tier 2.

A production Tier 1 classifier can be deployed through an ONNX (Open Neural Network Exchange) or TensorRT runtime. The controller logic around that model is still simple: inspect every category score, choose the strongest permitted block candidate first, otherwise route the strongest review candidate onward.

Threshold tuning is itself a product decision. Set block thresholds conservatively to avoid over-blocking. review thresholds can be looser since humans or Tier 2 are in the loop. Tune these thresholds against measured false-positive and false-negative rates.

This detail matters in production because moderation is a multi-label problem. A post can look borderline in one category and clearly violating in another. The controller needs to inspect all category scores before emitting a final action.

Treating buyer-seller dispute language the same as general social text creates false positives. A message like "This product is fake garbage, you scammer" can be a legitimate refund dispute from a wronged buyer. Include conversation context, order status, previous messages, and user history before auto-blocking.

Tier 2: Policy-aware judge (budgeted escalation path)

For content that is uncertain (for example, threats versus sports slang, or policy criticism versus targeted harassment), send a richer context package to a policy-aware judge model if the synchronous latency budget permits it; otherwise hold the action for asynchronous review. Candidate safeguard layers include Meta's Llama Guard line^[1], Google's ShieldGemma^[2], hosted multimodal moderation APIs such as OpenAI's omni-moderation model^[3], or an internal judge wrapped with strict output schemas.

The design split is operational: a deployed safeguard model scores the taxonomy and version it was tested against, while a policy-injected judge can consume a newer approved policy pack without retraining its weights. That does not make a prompt edit enforcement-ready by itself. New policy packs still need golden-case and adversarial evaluation, schema validation, approval, monitoring, and rollback.

Instead of baking every rule into model weights, we inject the current policy definitions directly into prompt or retrieval context. That makes the system policy-aware and lets many rule changes ship as configuration updates rather than full retrains. The code block below illustrates a system prompt containing the exact policy rules. It serves as the instruction template for the Tier 2 judge. The template takes the user's content as an input variable and instructs the model to output a structured JSON decision with a short policy rationale.

The judge response is untrusted model output until the controller validates its schema and attaches the policy version that produced it:

Advantages

• Policy agility: Release a tested, versioned policy pack faster than retraining a classifier.
• Contextual awareness: It can distinguish between "I want to kill you" (violence) and "I killed it at the gym" (slang).
• Auditability: It can emit a concise rationale and confidence score that help downstream review tools after validation.

Tier 3: Human review

When the judge returns ESCALATE, fails output validation, or cannot make a permitted enforcement decision, ordinary ambiguous content can enter human review. This is an expensive and slow path, but it supports remediation and labelled evaluation data.

Review routes and service-level objectives depend on the action and harm category. Some sensitive categories require restricted workflows and applicable reporting procedures rather than being displayed in a general moderation queue.

Reviewer safety tooling can include protected access, blurring by default, controlled media playback, workload rotation, and support resources. Review outcomes can become evaluation or training labels only through governed data handling and quality checks.

Multi-modal content

This marketplace includes text, images, video, and audio, so its policy system needs paths for every supported content type. The architecture extends the tiered approach by passing each medium through specialized signals before combining policy-relevant context.

A major engineering challenge in multimodality is handling cross-modal context. A benign text overlay like "Look at what I found today" becomes a policy violation when paired with a graphic image. On e-commerce platforms, a listing photo showing a generic handbag paired with text "100% authentic Louis Vuitton" is a counterfeit violation only visible when both modalities are judged together. To handle this, the outputs from individual classifiers need to feed into a late-fusion layer or a multimodal LLM that can evaluate the combined context of the post.

Here is how a keyframe sampler works in practice. Given a video, it extracts frames at a fixed interval (e.g., every 1 second) plus additional frames whenever a significant scene change is detected. The combined set of keyframes is then passed to the image classifier pipeline. Scenes with rapid cuts or flashing content need extra coverage.

Sampling supplies evidence; it does not decide policy alone. A fusion controller can hold a listing when two moderate signals jointly cross a review boundary:

For live streams, use a sliding window of the last N frames and continuously emit keyframes to the moderation pipeline. For on-demand video uploads, batch processing is cheaper because you can wait for the full file before starting inference.

Once the individual pipelines process their respective modalities, a calibrated fusion layer can combine their signals or provide context for a multimodal judge. Thresholds must be fitted on representative labelled content because raw scores from different models are not automatically comparable. If the combined signal falls within the uncertain range, hold the listing and escalate the relevant context to review.

Treating video as a black box hides risk. Moderating every frame is usually too expensive, so use keyframe sampling plus audio transcription. For flashing or rapidly cut content, raise the sampling rate or add scene-change triggers.

Handling policy evolution

A major challenge in moderation is that policy and abuse patterns change. A new counterfeit phrasing may need an urgent response. A versioned Tier 2 policy pack can often be evaluated and released faster than a retrained classifier artifact, but it must not bypass approval, regression cases, monitoring, or rollback. Tier 1 can receive an approved deterministic signature quickly when an exact known pattern exists; learned generalization still needs examples, threshold tuning, and deployment evaluation.

A Tier 2 prompt or retrieval update is a candidate release, not an enforcement shortcut. Run policy regression cases, check output schema and thresholds, approve the new version, and monitor its rollout.

Well-designed systems also continuously generate "red team" data to test classifiers against adversarial attacks (for example, using datasets like ToxiGen^[4] to validate resilience against implicit hate speech).

Scaling to 10K+ RPS

Building a system that can accurately classify content is only half the challenge. The other half is making sure the system stays responsive when traffic spikes unpredictably during major global events. Achieving 10,000+ requests per second (RPS) requires heavy optimization at the infrastructure layer. We can use a combination of caching, batching, and intelligent routing to keep latency low and compute costs manageable.

Content fingerprinting and caching

Duplicate and near-duplicate content is common (reposts, viral memes, copypasta, and mass-uploaded counterfeit product listings). A fingerprint layer can avoid repeated model work, but only when its reuse rule does not silently broaden enforcement.

An exact decision cache can reuse an approved result for identical content under the same policy version, enforcement scope, and content type. A SimHash-style locality-sensitive hashing (LSH) index can find near-duplicates, but similarity is weaker evidence: route the match to review or an additional validated detector instead of copying a block decision automatically. The runnable version below keeps stores in memory so you can see that distinction without external services.

Cache namespaces must include policy version and enforcement scope. Otherwise yesterday's ALLOW, or another region's BLOCK, can be applied under a different rule.

Here is why LSH still matters. If a reviewed abusive message is reposted with a small addition, an exact SHA256 match misses it while SimHash can retrieve the related prior case. A tuned Hamming-distance threshold trades recall for false-positive risk; the example uses a deliberately permissive threshold to expose the near-duplicate routing behavior, not to prescribe an enforcement threshold.

On repost-heavy surfaces, measure how often exact hits safely avoid inference and how often similarity retrieval improves review throughput without raising false blocks.

Here is how fingerprinting strategies differ in enforcement strength:

Latency and hit rate depend on index design and scale; benchmark them on the actual workload rather than attaching universal millisecond values.

Exact-cache savings can be large on repost-heavy traffic. Similarity hits are useful too, but should not turn approximate matching into unreviewed enforcement.

Micro-batching amortizes GPU launch overhead and memory transfers across requests. Its benefit and latency cost must be measured on the deployed model and traffic shape.

GPUs are throughput-optimized devices. Processing requests one by one can underutilize available compute. Micro-batching accumulates incoming requests into a bounded batch before running inference, so the GPU processes items together and shares overhead; the configured size and timer are workload choices, not universal constants.

This is distinct from training-time gradient accumulation. At inference time, each request remains a separate moderation decision. The table below is illustrative load-test data for reasoning about the tradeoff, not a benchmark promise:

The scheduler needs a timer as well as a maximum batch size. The following toy schedule emits a full batch promptly during a burst and expires a partially filled batch before its wait budget is exceeded:

Strategy

Two common accumulation strategies exist, each with a different latency-throughput trade-off:

1. Window-based: Wait a fixed time window (e.g., 5ms) regardless of batch size. This keeps tail latency bounded, but during low-traffic periods you may underfill the batch.
2. Size-based: Wait until the batch reaches a target size (e.g., 32). This maximizes GPU utilization, but very slow requests at the head of the queue block the rest.

Production systems commonly use a hybrid: start a timer when the first request arrives, and fire the batch when either the timer expires or the batch fills, whichever comes first. This bounds the scheduler's added wait time; end-to-end p95 or p99 still depends on queues, inference time, downstream review, and overload behavior.

The actual batching and GPU scheduling is usually handled by specialized inference servers. Triton Inference Server can host custom batching logic, while systems built around PagedAttention and continuous batching (such as vLLM) keep decode pipelines saturated under mixed loads^[5]. Both approaches help avoid the GPU starvation that occurs when requests are processed one by one.

Batching alone is not enough. Set per-request deadlines at the gateway layer, then define a risk-approved timeout action such as hold for review or suppress an individual message. Do not silently bypass a required moderation decision on timeout.

Geographic distribution and regional policies

Network distance contributes to latency, so synchronous chat moderation may benefit from classifiers deployed near request traffic. Slower review paths can use regional hubs when latency, data-residency, model availability, and cost requirements permit it.

Beyond latency, geographic distribution raises a policy-routing problem. Applicable duties can depend on where content is offered, the user/account market, the action being taken, and legal policy configuration. A reviewed policy-resolution service should produce the enforcement scope used by inference and auditing.

Policy routing strategy

• The gateway sends account market, content availability, product surface, and other approved signals to a policy resolver; an IP hint alone is not an enforcement rule.
• The resolver returns a versioned enforcement_scope, policy overlay, and appeal/reporting route. Tier 2 receives that approved overlay alongside the base policy.
• Tier 1 may use scope-specific thresholds only after evaluation on the relevant traffic; unknown scope routes to hold/review where an enforcement decision is required.
• Regional obligations vary. For example, the EU's Digital Services Act (DSA) provides statement-of-reasons and transparency mechanisms for covered moderation decisions^[6], while India's IT Rules include grievance-related intermediary obligations^[7]. Product counsel translates those requirements into policy configuration.

Architecture pattern

One deployment option is Tier 1 in high-volume regions and Tier 2 in fewer regional hubs, with synchronous escalation only where its measured latency fits the surface budget. Placement is an engineering and compliance decision; language quality, residency rules, reviewer availability, and measured traffic may lead to different regional layouts.

Appeal and review workflow

No moderation system is perfect. The cost of false positives and false negatives varies by category and action. A counterfeit-listing auto-block may require very high precision and a fast appeal path; apparent child sexual abuse material (CSAM) or credible imminent harm requires a restricted safety workflow and applicable reporting/escalation obligations. Thresholds and remediation therefore belong to policy, not to a single global precision-versus-recall rule.

When a user appeals a moderation decision:

1. Record lookup: Load the content, action, policy version, enforcement scope, evidence, and user-visible reason that produced the decision.
2. Independent re-evaluation: Evaluate under the same governing policy version, or explicitly record a new policy version if policy changed. A larger model or specialist can add context; it must not use a silently "lenient" rulebook.
3. Authorized review: Route unresolved ordinary disputes to the right review queue. Keep restricted categories inside their authorized safety process.

Automation can prioritize appeals and surface obvious mismatches, but the proportion resolved automatically is a measured product outcome, not an architectural assumption.

The feedback loop

The appeal system's real value is in generating training data:

• False Positives: These are gold. They become "hard negatives" for the next training round of the Tier 1 classifier, actively reducing future false ban rates. On LogiFlow, a falsely blocked luxury watch listing because of aggressive brand-protection thresholds is a critical false positive to feed back.
• Confirmed Violations: Added to the dataset as new positive examples to strengthen pattern recognition. A confirmed counterfeit electronics listing becomes a new positive training example for the image pipeline.
• Novel Violations: Edge cases that humans struggle with typically trigger formal policy reviews and prompt updates for the LLM.

Training moderation models on harmful content requires strict safety controls to prevent inadvertent exposure. In practice, teams combine carefully controlled real examples from review queues with synthetic and adversarial examples that broaden coverage without forcing every engineer to handle raw toxic data. For the most sensitive categories, systems should prefer hashes, signatures, and restricted-access review tooling over broad dataset access. Human annotators who do review raw content operate under rotation policies and have access to psychological support, limiting cumulative exposure. Verified decisions from the human review pipeline become ground-truth labels for offline retraining cycles, ensuring the automated system improves while raw harmful data stays tightly controlled.

Regulatory and cultural considerations

Content policies are rarely global. What's considered standard political discourse in one country might be illegal hate speech in another. A well-architected moderation system needs to be flexible enough to apply different rulesets based on user geography, while still maintaining a baseline of universal safety.

Implementation

• Base Policy: Global rules (e.g., no CSAM, no malware).
• Regional Overlays: Resolve a versioned enforcement scope from approved product and jurisdiction signals, then inject its instructions into Tier 2.
• Geo-fencing: Some content categories may be blocked in one region but allowed in another, so enforcement scope must be part of the decision record.

Mastery check

In a system design interview or production review, you should be able to explain:

• Why a cascade of hashes, rules, classifiers, LLM judges, and humans beats a single giant model.
• How versioned policy packs let Tier 2 change before Tier 1 is retrained without skipping evaluation and release controls.
• How exact hashes can reuse same-scope decisions while similarity hashes retrieve review evidence without copying enforcement.
• How micro-batching may improve GPU throughput while a timer bounds its contribution to p95 latency.
• How late fusion combines text, image, audio, video, user-history, and policy signals.
• How governed appeal outcomes can create hard negatives and high-value evaluation or training labels.
• How adversarial users exploit Unicode homoglyphs, character substitutions, encoded text, and cross-modal context, and why red-team suites need continuous updates.

Evaluation rubric

• Strong answer starts with the cascade and explains why LLM-only moderation fails on latency, cost, and throughput.
• Strong answer separates ALLOW, BLOCK, and REVIEW thresholds instead of treating moderation as one binary cutoff.
• Strong answer explains why a tested Tier 2 policy pack may release before Tier 1 retraining, while neither path bypasses approval, evaluation, and rollback.
• Strong answer includes scoped exact-cache reuse, similarity-to-review routing, batching deadlines, policy resolution, and appeal logging as first-class system design elements.
• Strong answer treats ordinary human review as due process and a governed label source, while routing sensitive categories through authorized restricted workflows.

Follow-up questions

1. A new counterfeit wording pattern appears today and policy must change before tonight's seller spike. Which versioned rule or policy-pack release can be evaluated first, and what classifier artifact changes later?
2. Your review queue doubles after a flash sale, but p95 chat latency still has to stay under 200 ms. Which threshold, routing, or batching knob do you inspect first?
3. A sports-fan message hits a violence review threshold at 0.68 with a block threshold of 0.95. Why is Tier 2 routing better than auto-blocking here?
4. One abusive meme is reposted 50,000 times with tiny text edits. Which fingerprint layers retrieve it, when may an exact decision be reused, and why should approximate matches enter review by default?
5. EU and US enforcement overlays disagree on a listing appeal. What fields must be present in the decision record to explain and audit the action later?

Common pitfalls

• Symptom: moderation costs explode and p95 latency misses target. Cause: every request reaches the LLM judge. Fix: expand evaluated deterministic or classifier paths where policy permits, then reserve Tier 2 for cases needing context.
• Symptom: legitimate seller listings are blocked after a policy update. Cause: a new Tier 2 policy pack skipped regression gates, or Tier 1 still applies an incompatible artifact. Fix: release a tested version with rollback, then retrain or retune the fast path with fresh appeals and misses.
• Symptom: old or merely similar content receives an automatic enforcement decision. Cause: cache keys ignore scope or near-duplicate retrieval copies an action. Fix: namespace exact decisions by version/scope and route approximate matches to review.
• Symptom: GPU throughput improves but chat feels slower. Cause: batching waits too long or lets one slow request hold the batch. Fix: use size-or-timer batching plus gateway deadlines and explicit fallback behavior.
• Symptom: refund disputes and counterfeit complaints get over-moderated as harassment. Cause: buyer-seller commerce language is treated like generic social chat. Fix: include order, listing, and conversation context before auto-blocking.

Summary

1. Tiered Architecture: Route high-confidence routine cases through validated fast paths; hold or escalate ambiguity and high-impact actions.
2. Latency vs. Risk: For the scenario's <200 ms chat target, benchmark exact-cache hits, classifier time, batching waits, and any synchronous escalation.
3. Policy Agility: Version and evaluate policy-pack releases before enforcement; update learned classifiers after labelled evidence is ready.
4. Multimodality: Don't ignore images/video. Use specialized pipelines and fuse the signals.
5. Feedback Loops: Governed review and appeal outcomes can improve evaluation sets and future models.

Common misconceptions

• "Just use an LLM for everything." It forces every request onto the most expensive contextual path. At the scenario's 10K RPS target, benchmark a cascade against any LLM-heavy proposal before selecting it.
• "Moderation is a binary classification problem." It's highly contextual. "I hate you" is bad; "I hate Mondays" is fine. Context matters.
• "Once trained, the model is done." Abuse patterns and policy change. The system needs monitored policy versions, evaluation updates, and retraining when labels justify it.

You have now designed a real-time content moderation architecture using tiered classifiers, policy-aware judges, scoped fingerprint caches, bounded micro-batching, policy resolution, and appeal/restricted-review paths.

The next capstone takes the same latency discipline and context hierarchy, then applies serving techniques such as KV-cache reuse and speculative decoding to real-time code completion inside an IDE.