Revenue Protected vs Conversion Loss — The Fraud ROI Metric That Matters
"Revenue protected" reports tell you only half the story. The other half — conversion loss to false positives — is what separates real fraud ROI from vanity metrics.
Most fraud-prevention reporting focuses on a single number: revenue protected. The fraud app surfaces a satisfying figure — "we prevented $42,000 in chargebacks this quarter" — and the merchant feels reassured.
The problem: that number tells you only half the story.
The other half — how much legitimate revenue was lost because of false positives — almost never appears in the same report. Without both halves, you can't tell whether your fraud prevention is making you money or costing you money.
This guide covers the conversion-loss measurement that pairs with revenue-protected reporting, why most stores under-measure it, and the framework for evaluating whether your fraud-prevention investment is actually positive.
What each metric actually measures
Revenue protected is the dollar value of orders that would have been fraudulent if your prevention hadn't fired. Methodology varies by fraud app, but generally:
- An order is flagged or cancelled based on fraud signals
- The system records the order value
- The system estimates this order would have resulted in a chargeback or fraud cost
- The recorded value rolls up into "revenue protected"
Conversion loss is the dollar value of legitimate orders that were prevented by the same controls — false positives where the customer would have paid, received, and not disputed.
Conversion loss is harder to measure because false positives are silent. The customer either doesn't reach out (most common), reaches out and gets manual override, or reaches out and accepts the rejection. The first category is invisible in standard reporting.
The full economic story:
Net value of fraud prevention = Revenue protected − Conversion loss − Operational cost of the prevention system
Why most stores under-measure conversion loss
A few specific reasons:
Fraud apps default to reporting only the favorable number. The "revenue protected" figure is easy to compute and looks good. The "conversion loss" figure is harder to estimate and looks bad. Most vendors highlight one and not the other.
False positives are invisible by default. The customer whose legitimate order was cancelled and who never reached out is, by definition, undetectable. Measuring conversion loss requires infrastructure to surface these cases.
Merchants don't track contact outcomes. When a customer reaches out about a cancelled order, the support interaction often doesn't get categorized as "false positive recovery." Data exists but isn't aggregated.
The math is uncomfortable. Calculating conversion loss often reveals that fraud prevention is destroying more value than it's protecting. Merchants who suspect this have an incentive not to look too closely.
Four methods to estimate conversion loss
Ordered by accuracy:
Method 1: Customer-contact tracking
For 90 days, every customer contact about a cancelled or blocked order gets categorized:
- True positive (customer admits or evidence confirms fraud)
- False positive (customer was legitimate, would have paid)
- Unclear (insufficient evidence either way)
Calculate the percentage of contacts that were false positives. Apply that percentage to total flagged-and-cancelled order volume to estimate false-positive rate. Multiply by cancelled-order value to get conversion loss.
Strengths: Direct measurement.
Weaknesses: Only counts customers who reached out. Silent abandonment isn't captured.
Method 2: Statistical sampling
Take a random sample of cancelled orders (say, 50/month) and proactively reach out with a brief verification message. Based on responses, classify the sample. Extrapolate to the full population.
Strengths: Captures silent-abandonment population.
Weaknesses: Operationally expensive. Response rate limits accuracy.
Method 3: A/B testing your fraud rules
For a defined period, randomize a subset of orders that would have been cancelled to be released instead. Track outcomes — disputes, chargebacks, returns. The release-group's actual fraud rate tells you the true positive rate of your rules.
Strengths: Highest-accuracy measurement.
Weaknesses: Requires accepting some fraud loss during the test. Some merchants are uncomfortable with this trade-off even though the data is valuable.
Method 4: Historical-pattern modeling
Look at orders that were flagged but not cancelled (because rules were less strict at the time, or because human review released them). Track outcomes. Calculate fraud rate of "flagged but not cancelled" orders. Assume cancelled orders have the same true-positive rate.
Strengths: Uses existing data, no operational disruption.
Weaknesses: Assumes cancellation rules have same accuracy as historical detection — may not hold.
For most stores, Method 1 (customer-contact tracking) is the most practical starting point. Method 3 (A/B testing) is most accurate when you can do it.
What the numbers usually show
When stores actually measure both sides for the first time, common patterns emerge:
Conservative rules
Revenue protected significantly exceeds conversion loss. Net positive. Most rules at this level are well-tuned.
Aggressive rules
Conversion loss approaches or exceeds revenue protected. Net negative or near-zero. Prevention is destroying as much value as it preserves.
Mixed rules
Some rules are conservative (high precision, modest scope); others are aggressive (broad scope, high false-positive rate). Aggregate net value is positive but specific rules are destructive.
The third pattern is the most common and the most actionable. It means there's a specific rule (or two) responsible for most of the conversion loss, and removing or tuning that rule would improve net value significantly without affecting the rules that work.
A working evaluation framework
For each major fraud rule, calculate:
- True positive rate: Of orders this rule fires on, what fraction actually were fraud?
- Revenue protected by this rule: True positives × average value × fraud cost multiplier
- Conversion loss from this rule: False positives × average value × (1 - margin)
- Net value: Revenue protected − conversion loss
A rule with negative net value is destroying revenue. A rule with marginal positive net value (e.g., $1.10 protected per $1 lost) is too close to call — small changes in fraud patterns can flip it negative. A rule with strong positive net value (e.g., $5+ protected per $1 lost) is durable.
The framework lets you make per-rule decisions: keep, tune, or remove.
What the metric should drive
The revenue-protected-vs-conversion-loss numbers should drive specific decisions:
| Situation | Action |
|---|---|
| Net value strongly positive, false-positive rate manageable | Keep operating. Look for additional patterns to capture. |
| Net value modestly positive, FP rate uncomfortable | Some specific rules likely net-negative. Identify and tune the worst offenders. |
| Net value near zero | Prevention system isn't adding value. Pull back to simpler controls covering only highest-confidence fraud. |
| Net value negative | Active destruction of revenue. Roll back aggressive rules. Focus only on controls with clear net positive evidence. |
The framing matters. Default merchant assumption is "more fraud prevention = better." Data sometimes says otherwise.
A specific example walk-through
Consider a store with this rule: "Auto-cancel all orders flagged as high-risk where IP geolocation doesn't match billing country."
Over 90 days:
- 1,200 orders matched the rule
- 300 confirmed fraud (chargebacks, customer admissions)
- 200 confirmed legitimate (customer reached out, manual review released)
- 700 unclear (no contact, no recurrence)
If we assume unclear cases have the same true-positive rate as contact-based sample (300/500 = 60%):
- 420 of 700 unclear = fraud
- 280 of 700 unclear = legitimate
Total: 720 fraud, 480 legitimate. False-positive rate: 40%.
If average order value = $120, average fraud cost (chargeback + product + processing) = $200:
- Revenue protected: 720 × $200 = $144,000
- Conversion loss: 480 × $120 = $57,600
Net: $86,400 positive. The rule is profitable but false-positive rate is high.
Next move: look for ways to tighten the rule (combine with additional signals) to reduce false positives without sacrificing fraud-prevention value.
This kind of analysis is what the dashboard "we protected $X this quarter" never enables.
Common pitfalls
Reporting only the favorable number. Stores that brief leadership on "revenue protected" without mentioning conversion loss give an incomplete picture. The conversation about fraud-prevention investment should always include both sides.
Not tracking by rule. Aggregate numbers hide which specific rules are profitable. Per-rule tracking is where actionable decisions live.
Discounting customer-lifetime impact. Conversion loss measured as "value of lost order" understates impact when the customer would have become a repeat buyer. For high-LTV businesses, true conversion loss is 2-5× immediate order value.
Ignoring brand impact. Some false-positive customers leave negative reviews or share their experience socially. Brand cost isn't always captured in conversion-loss calculations.
Treating the metric as static. Fraud patterns change. Customer base changes. Rule effectiveness changes. Revisit regularly.
How Shieldy surfaces this
Shieldy Fraud Filter's analytics dashboard reports both sides:
- Revenue protected per rule (with confidence interval)
- Conversion loss estimate per rule (based on customer-contact data Shieldy tracks)
- Net value per rule
- Aggregate fraud-prevention ROI
Shieldy also supports the A/B testing approach (Method 3) — configurable random-release-percent per rule, with outcome tracking that updates true-positive rates in real time.
A practical first-month exercise
If your store has fraud prevention running but you've never measured both sides:
- Pull 90 days of cancelled orders
- Cross-reference with customer contacts during that period — which cancelled orders had customer reach-out?
- Categorize contacts (true positive / false positive / unclear)
- Estimate false-positive rate based on categorized sample
- Apply rate to all cancelled orders to estimate total conversion loss
- Compare to your "revenue protected" figure
- Decide whether net is satisfactory
The exercise rarely produces the result merchants expect. Good news: even when the math is uncomfortable, knowing it lets you make better decisions.
A practical close
Fraud prevention's value isn't the fraud it stopped. It's the fraud it stopped minus the legitimate revenue it cost.
Shieldy reports both. Most fraud apps don't. The discipline of measuring conversion loss honestly — and tuning rules based on the net — separates merchants who get real ROI from prevention from those who buy peace of mind.
Protect your Shopify store today
Install Shieldy free — block fraud, bots, and VPNs in under 5 minutes.
Install on Shopify — Free
