User Feedback & Implicit Signals¶

TL;DR

• Capture both explicit and implicit signals. Log thumbs, reason chips, and inline corrections alongside behavioral signals like retries, regenerations, escalations, and copies.
• Link every feedback event to a trace_id. Attach user signals directly to trace records so you can inspect the exact prompt, context, and SQL query.
• Treat user feedback as a triage signal, not a rate. Use biased feedback to find failures, and rely on uniform-sample judges for unbiased quality rates.
• Promote feedback to dataset labels in stages. Corroborate user signals with LLM judges and SME review before adding cases to golden datasets.

• You will learn:
  • Explicit and implicit signal catalog
  • Sparsity and bias corrections
  • Feedback UX design
  • Trace linking, label promotion, and weekly error analysis
• Prerequisites: Observability & tracing, Online evals
• Related: Production data flywheel, SME golden datasets, Monitoring & drift, LLM-as-a-judge

1. Why feedback matters, and why it isn't enough¶

Online LLM judges evaluate whether an answer is grounded and relevant. Only real users know whether an answer is useful for their task, timeline, and level of detail. User feedback also provides your cheapest stream of real production failure cases for offline datasets.

However, raw user feedback has four major limitations:

• Sparsity: Most user sessions leave zero explicit ratings.
• Selection bias: Users who click feedback buttons do not represent your average user population.
• Ambiguity: A thumbs-down can mean wrong facts, high latency, bad formatting, a policy refusal, or frustration with company policy.
• Latency lag: Downstream signals (such as reopened tickets or disputed finance numbers) arrive hours or days after the conversation.

Use feedback to surface and prioritize candidate failures, and verify them with LLM judges and human reviewers before treating them as ground truth.

2. Explicit feedback¶

2.1 Feedback UX design checklist¶

• Collect feedback per assistant turn, not just per session. Session-level CSAT cannot pinpoint which turn failed.
• Make rating one click and elaboration optional. Capture 👍 or 👎 immediately, then display optional reason chips.
• Map reason chips directly to your failure taxonomy (see Agentic RAG failure modes), such as Wrong facts, Didn't find the doc, Wrong numbers/SQL, Did the wrong action, Refused unnecessarily, and Too slow.
• Enable inline user corrections on structured outputs such as SQL queries, financial figures, or ticket fields.
• Acknowledge submitted feedback immediately and notify users when their reports lead to fixes.
• Avoid dark patterns. Never block workflow progress with pop-ups or hide the 👎 button.
• Attach trace metadata automatically. Log trace_id, version tags, and UI state without asking the user to re-explain their prompt.
• Show a clear privacy notice. Inform users that human reviewers may inspect flagged conversations.
• Separate security and data-leak reporting from general quality feedback and route security flags to an on-call queue with an SLA.

3. Implicit signals¶

Implicit signals come from natural user behavior. Every session generates implicit signals, but you must interpret them carefully for your product workflow.

3.1 Turning raw events into signals¶

Convert raw clickstream events into structured per-trace and per-session signal records stored alongside your online eval scores:

# Requires Python 3.10+ (uses the `int | None` syntax).
from dataclasses import dataclass

@dataclass
class TurnSignals:
    trace_id: str
    thumbs: int | None          # +1, -1, None
    reason: str | None          # chip value
    regenerated: bool
    rephrased_next: bool        # next turn semantically similar within 60s
    negative_followup: bool     # classifier on next user turn
    copied: bool
    escalated_within_30m: bool
    ticket_outcome: str | None  # "resolved", "reopened", "rejected", None

def implicit_failure_score(s: TurnSignals) -> float:
    """Heuristic triage score in [0, 1]; weights are starting points to calibrate, not truths."""
    score = 0.0
    score += 0.5 if s.thumbs == -1 else 0.0
    score += 0.3 if s.negative_followup else 0.0
    score += 0.25 if s.regenerated else 0.0
    score += 0.2 if s.rephrased_next else 0.0
    score += 0.3 if s.escalated_within_30m else 0.0
    score += 0.3 if s.ticket_outcome in {"reopened", "rejected"} else 0.0
    score -= 0.2 if s.copied else 0.0
    score -= 0.3 if s.thumbs == 1 else 0.0
    return max(0.0, min(1.0, score))

Calibrate these weights against human judgment. Hand-label ~200 traces as good or bad, fit a logistic regression model on the signal features, and recalibrate whenever UI layout changes alter button click rates.

3.2 Detecting rephrase and "that's wrong" follow-ups¶

• Rephrase detection: Embed consecutive user turns in the same session. Flag a rephrase when cosine similarity exceeds $\tau$ and the second turn arrives within 60 seconds. Calibrate $\tau$ on 200–300 labeled turn pairs so "Show Q3 EMEA revenue" → "EMEA revenue for Q3 FY26 please" triggers the flag, while "Now break it down by country" does not.
• Negative follow-up detection: Run a lightweight classifier or small LLM on the next user message to classify correction, frustration, clarification, new_topic, or thanks. This signal catches natural-language complaints across languages without requiring button clicks.

4. Biases and sparsity¶

4.1 Common biases¶

As Joachims et al. demonstrated in "Accurately Interpreting Clickthrough Data as Implicit Feedback", click data works best as a relative preference signal due to strong position bias. Apply the same principle to citation clicks.

4.2 Dealing with sparsity¶

• Report feedback coverage: Compute feedback_rate = traces_with_explicit_feedback / total_traces and display it next to every feedback chart.
• Aggregate over sufficient volume: Compute 👎 rates over weekly or biweekly windows with confidence intervals rather than noisy daily slices.
• Combine explicit and implicit signals: Use the composite implicit_failure_score (Section 3.1) to rank all traces, not just the small share with thumbs (often low single-digit percent; measure your own feedback rate).
• Avoid running A/B tests on thumbs alone unless power calculations confirm sufficient sample size (see Online evals §5.3).
• Solicit targeted feedback sparingly: Prompt users for ratings slightly more often on canary versions or new intents, and tag those events solicited = true.

5. Linking feedback to traces¶

User feedback is useless for debugging unless you can join it back to the exact execution trace.

sequenceDiagram
    participant U as User
    participant UI as Chat UI
    participant API as Assistant API
    participant T as Trace store
    participant FB as Feedback service
    U->>UI: asks question
    UI->>API: request
    API->>T: spans with trace_id
    API-->>UI: answer + trace_id
    U->>UI: thumbs down + reason "Wrong numbers"
    UI->>FB: feedback event with trace_id, session_id, reason
    FB->>T: attach score "user_feedback" to trace_id
    T-->>T: trigger async judges and review queue

Follow this trace-linking checklist:

• Return trace_id and session_id with every API response and store them on the frontend message object.
• Include trace_id, session_id, user_id_hash, timestamp, type (thumbs, rating, comment, correction, implicit:*), value, reason, ui_version, and solicited on every feedback event.
• Log external correlation IDs (such as ticket_id) on tool spans so delayed webhooks can join back to the originating trace_id.
• Write feedback events both as trace scores in your observability platform and as rows in your analytics warehouse.
• Support feedback edits (such as changing 👎 to 👍) by updating the active score while preserving an audit log.
• Run free-text feedback comments through the same PII redaction pipeline used for traces.

6. Converting feedback into labels¶

Treat raw user feedback as a weak signal and promote it into trusted dataset labels through five stages:

Follow these rules when promoting feedback into dataset rows:

• Verify user corrections before trusting them. User edits provide fast reference answers, but SMEs must confirm the user was not mistaken or viewing restricted outside data.
• Harvest positive feedback too. Verified 👍 traces provide essential "must continue to pass" test cases that prevent your dataset from containing only edge-case failures.
• Never fine-tune or optimize prompts directly on raw thumbs. De-bias and review feedback first so you do not reward sycophantic or confident-sounding hallucinations.
• Feed judge disagreements into judge calibration. When SMEs confirm a 👎 trace failed despite a supported judge verdict, add that trace to your judge calibration set (see LLM-as-a-judge).
• Log label_source and confidence on every dataset row (user, judge, or sme).

7. Using feedback to prioritize error analysis¶

User feedback tells you where to look, while error analysis tells you what to fix. Run this weekly error-analysis loop:

flowchart TD
    A["All traces this week"] --> B["Rank by implicit failure score, thumbs down, judge failures"]
    B --> C["Review queue: top N plus random sample"]
    C --> D["Annotate: failure category, root-cause component, severity"]
    D --> E["Pivot: count by category x segment x version"]
    E --> F["Pick top 1-3 issues by impact = frequency x severity x reach"]
    F --> G["Fix: retrieval, prompt, SQL schema docs, tool, guardrail"]
    G --> H["Add regression cases to datasets"]
    H --> I["Verify offline, ship via canary, watch online signals"]
    I --> A

Follow these practices during weekly review sessions:

• Mix a 20% uniform random sample into your review queue. Reviewing only feedback-ranked traces blinds you to silent failures where users trusted a confident hallucination.
• Tag the failing component on every reviewed trace (query_rewrite, retrieval, rerank, generation, sql_generation, tool_selection, guardrail_false_positive, data_quality, policy_gap) so the right engineering owner can fix it.
• Start with open coding before locking your taxonomy. Write short plain-text failure notes on your first 50–100 traces, then group recurring patterns into taxonomy categories (see Hamel Husain and Shreya Shankar in Further reading).
• Rank fixes by impact = frequency × severity × reach. Weight severity highest for data leaks and financial errors, and lowest for minor formatting issues.
• Verify that shipped fixes move the target signal. After deploying a fix, confirm that the specific reason chip and implicit_failure_score drop for the affected user slice.

7.1 Example: from feedback to fix¶

1. Detect the spike: The 👎 rate with the Wrong numbers reason chip doubles for users with the finance role over seven days.
2. Isolate the slice: Filtering those traces reveals that all failing queries mention a quarter, and the online SQL-alignment judge flags 40% of them.
3. Diagnose root cause: SME review shows the generated SQL used DATE_TRUNC('quarter', …) (calendar quarters) instead of joining the fiscal calendar table, because the data engineering team renamed the fiscal calendar warehouse table without updating the SQL prompt schema description (see Monitoring & drift).
4. Ship the fix: Update the SQL prompt's schema documentation and add a deterministic SQL check requiring the fiscal calendar table join whenever a query mentions quarters.
5. Lock in regression coverage: Add 25 SME-reviewed queries with gold SQL to the text-to-SQL benchmark, roll out via canary, and confirm the Wrong numbers chip rate returns to baseline.

8. Metrics to report from feedback¶

Feed these metrics directly into the operational dashboards covered in Monitoring & drift.

Key takeaways¶

• Capture both explicit signals (thumbs, reason chips, inline edits) and implicit signals (rephrases, regenerations, negative follow-ups, escalations, ticket outcomes, copies) linked to trace_id.
• Treat explicit feedback as sparse and biased. Use feedback to discover failures, and use uniform-sample LLM judges to measure population quality rates.
• Map feedback reason chips directly to your failure taxonomy with one-click submission.
• Promote raw feedback into dataset rows through judge corroboration and SME review, logging label_source and confidence.
• Include a uniform random sample in every human review queue so you catch silent hallucinations that users never report.
• Treat free-text feedback as untrusted user input: redact PII, rate-limit submissions, and never inject raw feedback into prompts.

Further reading¶

• Hamel Husain, A Field Guide to Rapidly Improving AI Products and Your AI Product Needs Evals
• Shreya Shankar, Data Flywheels for LLM Applications
• Shankar et al., Who Validates the Validators? (criteria drift, aligning evaluators with human judgment)
• Eugene Yan, Patterns for Building LLM-based Systems & Products (section on collecting user feedback)
• Joachims et al., Accurately Interpreting Clickthrough Data as Implicit Feedback
• Radlinski, Kurup, Joachims, How Does Clickthrough Data Reflect Retrieval Quality?
• Within this repo: Production data flywheel, Online evals, Observability & tracing