A trading signal is a claim about the future. The only way to know if a claim-generator is any good is to keep score — and almost nothing in this space does that in public.
The question every AI trading tool avoids
Ask any AI stock-picking service one question: "Out of the last 100 signals you sent, how many were right, and by how much?" Most can't answer it. The signal feed and the scorecard live in different systems, if the scorecard exists at all.
That's the gap that matters. A model that's right 55% of the time with disciplined position sizing can be very profitable. A model that's right 80% of the time on rare setups it barely finds can be useless. Without a paired signal → outcome record, you can't tell the difference — you're trusting the vendor's word for it.
What a real verification record looks like
At minimum, look for:
- Every signal timestamped before the outcome — not "we called this" written after the fact.
- A stated outcome window — 24h, 72h, 7 days — decided in advance, not cherry-picked per trade.
- Calibration, not just accuracy — if a tool says "high confidence" on a call, high-confidence calls should win more often than low-confidence ones. If confidence doesn't correlate with outcome, the confidence label is decoration.
- Losses included — a track record with no losing calls has been filtered, not achieved.
Why this is harder than it sounds
Backtested accuracy and live accuracy diverge for reasons that have nothing to do with the model being "wrong":
- Regime drift. A signal tuned on 2023-2024 data can quietly stop working when volatility or correlation structure shifts — the model doesn't know it's stale.
- Survivorship in the review process. If a human curates which signals to publicize, the published record isn't the model's real hit rate.
- Outcome-window gaming. A signal graded "correct" if the price moves favorably at any point in a wide window looks a lot better than one graded on a fixed horizon.
Reading a reliability diagram
The simplest test of calibration is a reliability diagram. Group every signal by its stated confidence, work out how often each group was right, and plot stated confidence against observed hit rate. A calibrated model's points sit on the diagonal. Points below it mean the model is overconfident: its "high confidence" calls win less often than the label claims.
Three things make the chart honest:
- Count per bucket. A bucket with nine signals can swing wildly on one or two results. Show the count next to each point and merge sparse buckets.
- A fixed outcome window. Every signal is graded over the same horizon, or the hit rates are not comparable.
- Losers in every bucket. A high-confidence bucket with only winners is a sign of filtering, not skill.
Calibration says whether confidence means something. It does not say the model makes money. Pair the diagram with a score that penalises confident wrong calls, such as the Brier score, and with the full record. For a worked example and how to build one from your own signal log, see how to read a reliability diagram for AI signals.
The practical checklist
Before trusting a signal feed with real capital, ask for (or go look for) three things: a full history including losers, a fixed grading window applied uniformly, and evidence that stated confidence actually predicts outcome. If a provider can't produce that, you're not evaluating a strategy — you're evaluating a marketing page.