What is the difference between in-sample and out-of-sample?
In-sample data is the history used to build and tune a strategy, and out-of-sample data is history the strategy never saw during that process, held back so the finished rules can be graded on something they were not shaped by.
The distinction matters because a rule with enough adjustable parts can be made to fit any stretch of history. Fitting is not evidence of an edge; it is evidence that the rule is flexible. The only way to separate the two is to ask how the rule performs on data it had no opportunity to fit.
An out-of-sample result is therefore a different kind of number from an in-sample one. The in-sample figure tells you how well the rule can describe the past. The out-of-sample figure is the first estimate of how it behaves on the future, and it is nearly always worse. The size of that gap is itself informative.
Most published backtests report only the in-sample figure, sometimes without saying so. A result with no stated split should be read as in-sample until proven otherwise.
How should you split the data?
Split chronologically, hold back the most recent segment, make it large enough to contain a meaningful number of trades, and leave a gap between the two segments so that trades open at the boundary do not straddle it.
Chronological is not negotiable. A random split of bars or trades leaks the in-sample regime into the out-of-sample segment, because adjacent bars are highly correlated and the strategy has effectively seen the neighbours of every out-of-sample bar. Time-ordered splitting is the only arrangement that resembles what happens when a strategy is actually traded.
The proportion is a trade-off between a well-tuned rule and a meaningful grade. Holding back a fifth to a third of the history is common. What matters more than the fraction is the trade count in the held-back segment; an out-of-sample period with fifteen trades cannot distinguish an edge from noise whatever its length in years.
Regime coverage is the check people skip. If the whole out-of-sample segment falls in one kind of market, the grade only says how the rule handles that kind. Where the history allows, the held-back period should contain more than one regime, or the result should be reported as conditional on the one it contains.
What invalidates an out-of-sample result?
An out-of-sample result is invalidated by any decision made after looking at it, because the moment the held-back segment influences the rule it has become in-sample data with a misleading label.
The most common way this happens is iteration. A strategy is tuned, graded out of sample, found wanting, adjusted and graded again on the same held-back segment. After a few rounds the parameters have been selected for their out-of-sample performance, which is precisely the selection the split was meant to prevent. The segment has been spent.
The subtler version is selection among strategies. Test ten candidate rules, grade each once on the held-back segment, and keep the best. No single rule was iterated, yet the winner was chosen by its out-of-sample score, and the expected out-of-sample performance of the best of ten random rules is comfortably positive. This is the multiple-testing problem, and a split alone does not solve it.
The remedy is procedural. Decide in advance what result would count as a pass, grade once, and record the outcome whether or not it is the one you wanted. If the rule needs changing, the next grade needs history that has not been looked at yet, which is why a decade of data is not a luxury.
Is one split enough?
One split gives one grade on one period, which is far better than none and still a small sample of periods. Rolling the split forward through history turns one grade into many, and the consistency across them is a stronger statement than any single figure.
That rolling procedure is walk-forward optimisation. The in-sample window tunes, the following out-of-sample window grades, both slide forward, and the strategy's record is the concatenation of the out-of-sample segments. A rule that passes one split may have been lucky in that one period; a rule that passes most windows across a decade has been graded on many futures.
The rolling version also exposes something a single split hides: how much the tuned parameters move from window to window. Parameters that jump around are describing the noise in each window rather than a stable feature of the market, and that is visible only when there are several windows to compare.
None of this changes the rule about spending. Every out-of-sample window is still a one-look resource. Rolling the split forward creates more of them; it does not make any of them reusable.
How does QuantParadox apply the split?
QuantParadox applies the out-of-sample split by default and reports the held-back result separately, alongside the number of trades it rests on, rather than offering the split as an option that a hurried test can skip.
The default matters because the failure mode is not people choosing to skip the split. It is people forgetting, or running one more quick test on the full history to see what happens. Making the split the path of least resistance removes the decision.
The limitation is one of arithmetic rather than design. A rule that fires rarely, or a history that is short, leaves a held-back segment with too few trades to say anything, and the platform reports that condition rather than a figure that looks like a grade. A result that rests on twelve trades is presented as not established, not as a pass or a fail.
The rolling walk-forward procedure is available for the strategies that generate enough trades to support it, and it is the version to prefer when the question is whether the rule is stable rather than whether it worked once.