How to choose the "best" system ?

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

How to choose the "best" system ?

Postby rjay » Sun Sep 24, 2006 9:10 am

Given the choice, which would you have more confidence in:

a) a system that back-tested (over 30 months) with a 30% return and forward tested (over 12 months) with a 50%+ return


b) a system that had near identical back and forward testing returns of 35%

In other words, with all other things (such as no.of trades, % in market, etc.) being equal, is consistency the most desirable thing ?
Posts: 116
Joined: Wed Jul 26, 2006 6:51 am

Postby Overload » Sun Sep 24, 2006 12:16 pm

While pondering your question, I went back and forth a bunch of times. First I didn't think there was a preference at all, then I thought the 50%, and then I thought the 35%. In the end, it probably all depends how you look at it, not to mention what you want in a system. As well, the available information on the 2 systems is pretty limited, so there are a number of assumptions being made here. With that in mind, here's what I eventually settled on:

I'd go with the 50%+ return.

All systems have a degree of variability over time. In other words, we can say that the expected return for a system is X%, plus or minus Y%. But the truth is, when we back test a system, we don't really know if that back test was the absolute best the system could do, the absolute worst the system could do, or somewhere in the middle.

Let's say that in reality, System #1's returns might ultimately range from 5% to 35%. Our back test, however, showed a return of 35%. Thus, while our back test may have shown 35%, we can really expect returns more in the range of 10% to 30%.

But let's say that in reality, System #2's returns might ultimately range from 25% to 55%. Our back test for System #2 also showed 35%. Thus, while the back test was the same, the back tested return was at the low end of the spectrum of returns versus System #1, which was at the high end of the spectrum.

Connecting this to your 2 systems of (a) and (b), we still don't know where (b) lies within the spectrum of returns. It might be the case that 35% is the very top that this system will ever do, and the normal returns might be more in the range of 15% to 25%. But with system (a), it's already proven that the back test is not at the upper end of the spectrum. In fact, it's already proven that the upper end of the spectrum is 50%+. And that is a good sign.

But, like I said, this is just one way to look at it. As you know, there are many factors that come into play when considering something like this.

Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Postby Dacamic » Mon Sep 25, 2006 12:48 pm

A very good question ...

Boiling the subject scenario down to its simplest form, I would prefer trading the 35% / 35% system. A couple thoughts surfaced, however, while thinking about your question:
    1) Consistency is a desirable characteristic for trading systems, because a system with consistent performance will be predictable. In such case, we will trade with more confidence and easily monitor system performance during live trading.

    2) When evaluating consistency, we also need to consider the nature of our systems. For example, a "short only" system might sit quietly during strong uptrends, and then wake up during cyclical downturns. For such a system, back test performance might show high returns, yet forward test performance is a flat line. If the system performs consistently during its active periods and suffers minimal drawdowns in less ideal market conditions, we still have the consistency and predictability we desire.

    3) Trading the two example systems you described is not a mutually exclusive situation. I would scratch my head for at least a few moments trying to figure out a way to combine the two systems into one. I wouldn't spend a lot of time on this task, but maybe a little effort could create a combined system that produces a higher return and greater consistency than the two systems could on their own.
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm

Return to Curve-Fitting and Other Pitfalls