Scoring fraud in legal intake calls

The problem

I worked on RISQ, a fraud detection system for legal intake calls. Mass tort law firms get thousands of calls from potential claimants, and a significant share of them are fraudulent. Some callers are coached and read from a script. Some never used the drug or product in question. Some call again and again under different names.

RISQ listens to each call, scores it for authenticity and recommends one of three outcomes: transfer the caller to a closer, flag the call for review, or quarantine it.

The RIQ score

The core of RISQ is a composite score called RIQ, short for Real-time Integrity Scoring for Qualification. It is built from four scores, and three of them go into the composite.

RIQ = 0.20 * R + 0.35 * I + 0.45 * Q

The S-Score stays out of the composite because it measures the intake agent. A poor S-Score means the intake agent did not ask the right questions, so the other scores may be unreliable. It works as a pre-gate: if session quality is too low, RISQ flags the call for re-screening and makes no disposition.

Disposition logic

The scores feed a five-level disposition system. Each gate runs in order:

1. DQ check. Instant disqualification if the call violates campaign rules.
2. S-Score pre-gate. If session quality is critically low, the call is marked INCOMPLETE.
3. Hard gates. If the I-Score or Q-Score falls below an absolute threshold, the call is quarantined or flagged.
4. Checklist-only detection. Catches callers who hit every checklist item but show no genuine recall, which is a sign of coaching.
5. RIQ disposition matrix. The composite score maps to transfer, review or quarantine.

The checklist-only check is the one I find most interesting. Coached callers often have high Q-Scores because they studied what qualifies them. Their R-Scores are low because they cannot remember real details about an experience they did not have. A high Q-Score next to a suspiciously low R-Score triggers extra scrutiny.

The analysis pipeline

Each call goes through several stages.

1. Transcription and speaker diarization

AssemblyAI transcribes the call and separates the speakers. The caller's answers have to be isolated from the agent's questions, or the scoring is inaccurate.

2. Claim extraction

Claude Sonnet reads the transcript and extracts structured claims: which product the caller used, when, what symptoms they had, which doctors they saw and which facilities treated them. Each claim is tagged with a confidence level.

3. Linguistic fraud indicators

The same LLM pass flags language patterns associated with fraud:

• Unnaturally specific dates and details. Coached callers memorize exact dates but cannot recall the context around them.
• Legal terminology a regular person would not know, used repeatedly.
• Contradictions between the early and late parts of the call.
• Answers that sound scripted.

4. External verification

Extracted claims are checked against real databases:

• NPI Registry: confirms that named doctors exist and practice in the stated specialty.
• CMS database: confirms that named facilities are real healthcare providers.

5. Image verification

Some campaigns ask callers for photo proof, such as medical records or product packaging. RISQ sends an SMS requesting the photo, receives it through Twilio or Telnyx webhooks, and analyzes it with Google Gemini Vision to check that it matches the claimed evidence.

We called this stage NotHotDog, after the app in Silicon Valley. Its job is to decide whether the uploaded image is what the caller says it is, or a stock photo, someone else's medical record, or something unrelated.

Campaign rules as configuration

Each mass tort campaign has its own qualification criteria. A pharmaceutical case requires proof that the caller used a specific drug during a specific time window. A device recall case requires proof that the caller had a specific device implanted.

Instead of hardcoding these rules, RISQ reads a configuration per campaign that defines:

• Which qualification questions are required
• Which answers disqualify a caller
• Severity weights for each fraud indicator
• Which external databases to check
• Minimum score thresholds for each disposition

When a new campaign launches, the legal team writes the rules and the scoring engine applies them with no code changes.

What I learned

Fraud detection is adversarial, and fraudsters adapt. Once we started flagging callers with perfect checklist answers, some began getting minor details wrong on purpose to sound more natural. The scoring model needs regular recalibration.

Combining signals beats relying on one. Transcript analysis alone catches most fraud. External verification catches the more sophisticated cases. Image verification catches the rest. Each extra signal adds less on its own, but together they compound.

The S-Score was the most valuable addition. Measuring agent quality and gating on it prevented more false positives than anything else. A weak intake agent who asks leading questions can make a legitimate caller sound coached. Without the S-Score gate, those callers would have been wrongly quarantined.