Using out-of-sample data as a tool

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

Postby Dacamic » Fri Sep 01, 2006 10:53 am

The longer a system is active, the more likely it is to exceed its prior maximum drawdown. Plunging to such depths isn't necessarily a sign that a system has lost its edge; it is, however, statistically inevitable given enough time.

We ponder a plethora of performance values before we activate a system. The same should be true when wondering whether a system has reached its expiration date. During such deliberations, I try to answer one question:
    -- If I wasn't already trading this system, would I be willing to begin doing so today based upon it's historical performance?
If the answer is "yes", then I would continue trading a system even though it may have recently drooped to a historically low maximum drawdown.
Steve
Dacamic
 
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm

Postby rjay » Fri Sep 01, 2006 2:51 pm

I think I'm starting to see the point of archiving multi-systems as new ones are found .... what I'm thinking I should be doing is forward-testing the top 10 (or whatever) systems that were found. It's quite possible, even likely, that the top-performing back-tested system will not be the top-performing forward-tested system.

So - which takes precedence ? I'm leaning towards the best forward-tested system (best in all factors, not just the score or % return) because there's no possibility of curve-fitting.

Is the above what you folks are doing or do I have it completely wrong ?

WOW- I never knew I could find maths this interesting.
rjay
 
Posts: 116
Joined: Wed Jul 26, 2006 6:51 am

Postby Overload » Fri Sep 01, 2006 5:23 pm

Well first there's some disagreement that forward-testing is actually a good thing. Some think that if you have the data available, it's better to include it in your back test in the first place. This came out as I investigated the possibility of a forward-testing analysis being implemented into a OneClick Search. Here's the thread:

http://www.stratasearch.com/forum/viewtopic.php?t=88

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby rjay » Sat Sep 02, 2006 7:06 am

OK, yes I remember reading that thread.

Forward-testing gives me confidence to trade a system. Whether that is misplaced confidence or ignorance is debatable I guess. The biggest obstacle to successfully trading a mechanical system is ourselves so confidence is vitally important.
rjay
 
Posts: 116
Joined: Wed Jul 26, 2006 6:51 am

Postby Overload » Sat Sep 02, 2006 10:12 am

I am much the same way. In fact, I think I may have mentioned in that thread that I like to watch a system for a bit prior to trading, even though I know there is no technical (analytical?) benefit to doing so. As you said, it helps create confidence.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Previous

Return to Curve-Fitting and Other Pitfalls