Any other curve-fit tests?

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

Any other curve-fit tests?

Postby Overload » Wed May 16, 2007 11:04 am

Those that have looked at the current beta release will quickly see that there's been some pretty large emphasis placed on avoiding curve-fitted systems. In particular, we've added 3 pretty helpful features:

1) The ability to test and view surrounding parameter sets.

2) The ability to test and view alternate data sets (i.e. out-of-sample data)

3) A list of all the curve-fit protection mechanisms available, and whether those mechanisms are being used or not in the current setup.

StrataSearch has always provided the most common methods for avoiding curve-fitted systems. These include a large number of performance numbers like standard deviations that help identify consistency across trades and over time. But with these two new tests, I'm curious just how much more can be done.

The implementation of the parameter shift analysis flipped on a light bulb for me. Rather than thinking about over-fitting as an inherent flaw in the trading system itself, I began to think of over-fitting as simply nothing more than a lack of robustness in the system. Why would a trading system ever fail moving forward? The answer is always that that the trading system isn't robust enough to handle the revised data. You could take the most curve-fitted system possible and it would still perform well if the data were identical to the back test. But if even the slightest change in data occurs, such a system would fall apart. This is an example of an extremely un-robust system. But even a system that performs wonderfully across a wide range of tests may still fail moving forward. Why? Because it's virtually impossible to test every market scenario. A trading system can only be so robust. Every trading system exists somewhere in the robustness spectrum.

Taking this one step further, there is actually a balance that needs to be reached between robustness and returns. It is not possible for a trading system to be optimized for both at the same time. For example, if one wants to find a system that is robust enough to handle the crash of 2001-2002, it won't be able to maximize the returns of the rally from 2004 to present. A trading system simply can't be optimized for both, so a balance needs to be reached. And this is of course a personal decision, as a certain amount of guesswork needs to be done regarding upcoming market conditions.

Regardless of my own personal thoughts on this, I'd like to make an open-ended invitation for anyone to offer their thoughts and ideas on what else might be done to avoid curve-fitted systems. Without any checks for robustness, we know that StrataSearch can create systems with jaw-dropping returns. But we also have extensive tests in place to check for robustness and avoid the systems that don't meet our requirements.

Are there any additional tests that can be done? If you were visually examining a trading system, even with another piece of trading software, what else might you look at to determine whether the system would work moving forward or not?

The objective of StrataSearch is to not only build trading systems automatically, but to reject the non-robust systems automatically as well. That's a critical piece of the puzzle. And it seems to me that StrataSearch is now doing as much in this regard as any other system. If not, what else is there?

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby taowave » Sat Jun 02, 2007 9:33 pm

Hi Pete,
the addition of the alternate data performance test is really a major enhancement.the only thing I can think of is having a correlation tool for those who combine different systems into a multi.As I have said in the past,it is very insightful to see if one is combining highly correlated systems...or hopefully not if on the same side of he market...
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Sun Jun 03, 2007 8:47 am

Thanks, I really appreciate any input and ideas here, but maybe you can help me understand this one a bit better. How would having highly correlated strategies in a multi-system lead to curve-fitting? Are you thinking this would be an indication of a lack of diversity in the multi-system?

Thanks,
Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby taowave » Sun Jun 03, 2007 9:48 am

Overload wrote:Thanks, I really appreciate any input and ideas here, but maybe you can help me understand this one a bit better. How would having highly correlated strategies in a multi-system lead to curve-fitting? Are you thinking this would be an indication of a lack of diversity in the multi-system?

Thanks,
Pete


Its two fold.One part is definetly a lack of diversity.One of the savviest people I know who runs a major "market neutral fund" looks at two factors when hiring potential traders.The first is risk adjusted return,and more importantly how the particualr strategy correlates with his current portfolio.If the trader presents a great strategy that is highly corrrelated with the firms current book,he will have a very dificult time "selling" his system.What typically happens is the head of risk runs daily stress tests under adverse market conditions,and strategies that are negatively correlated to the firms position/orientation are the one that the fund manager seeks to add to the firms portfolio.I know this for a fact as I have presented an "option dispersion" strategy which had high correlation to the firms start arb models on the downside,and it was not something they wanted to take on at the current time.

Which brings us to my previous concern with multisystems.IMHO,one has to be very sure that a great performing multi system isnt fitted to the current market enviorment. Obviously,having an RSI system,a stochastic Xover,and a BBL x multi system is not what I would call diversified,and the returns of the systems are likely to be very correlated.Should the Multi be highly profitable in the current test period(perhaps due to an oscillating market),one should consider adding a breakout/moving average x over (opposite direction),or even a long option volatilty position for diversification purposes.Its really no diferent than running a portfolio and looking to add strategies by hiring traders.The ideal scenario is to hire successful traders who have negatively correlated strategies.

While you currently do not have a corelation matrix illustrating correlation of returns for strategies,you do have the best measures to insure robustness.MC,walk foward analysis,alternate data/perid are the best tools I have seen.I would be curious to see the correlation of returns when i run an OC for Highest Sharpe ratio.IMHO,a multi with negtively correlated strategies is easier to sleep at night than with multi with higly correlated strategies.

Does any of this babble make sense??
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Sun Jun 03, 2007 11:01 pm

Thanks, yes, that's a great explanation. And I do think you're onto something there. The benefits, from what I can gather, would primarily be from the additional diversity that comes from getting new strategies that are significantly different from existing ones. But I believe that's a worthwhile goal.

A certain degree of this "required diversity" is already included indirectly in an automated search. In particular, the scoring algorithm (if set up properly) will add strategies that most benefit the other strategies in the multi-system. For example, if the strategies you already have put you in the market 60% of the time, there would be little benefit (to the scoring) to give you additional trades covering that same 60% that you're already in the market. The only way you'll be able to improve your score is to find a strategy that gets you in the market during that remaining 40% of the time. And to do this, the search will need to find strategies that are significantly different from the ones already in place.

Nevertheless, there's no current way to actually verify just how diverse the strategies are, so your idea does make sense as a way to provide that verification. Thinking about it from a developers perspective, it's unfortunately a pretty big job. The display of the matrix would be quite simple, but we'd need to find some way to store the equity lines for each of the individual strategies. Numbers for the individual strategies within a multi-system aren't currently stored, or even calculated, so we're a ways away from having such data even available.

I guess the 2 formulas we currently have available are the cor() and alpha() formulas. These can both already be used to measure correlations with existing indexes (like the S&P 500 or DJIA). But they can't currently be used to measure a correlation against the equity line of an existing system.

I'll have to think about this a bit more. Perhaps if there were some way to export the equity line of an existing system into an Index, that Index could then be used within a cor() formula in a future strategy. If the cor() and alpha() and beta() numbers were then placed in the Summary tab or Performance Report, you'd at least be able to examine this manually, one strategy at a time. And the approach probably wouldn't be that hard to implement.

Anyway, sorry, that's my own share of babble right back at you. I like the idea. Something like this probably won't happen soon, but I'm going to continue to roll it around.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby taowave » Mon Jun 04, 2007 8:27 am

A certain degree of this "required diversity" is already included indirectly in an automated search. In particular, the scoring algorithm (if set up properly) will add strategies that most benefit the other strategies in the multi-system. For example, if the strategies you already have put you in the market 60% of the time, there would be little benefit (to the scoring) to give you additional trades covering that same 60% that you're already in the market. The only way you'll be able to improve your score is to find a strategy that gets you in the market during that remaining 40% of the time. And to do this, the search will need to find strategies that are significantly different from the ones already in place.


Up to now I have been keeping the criteria for OC's very simple.Essentially,I want strategies with high returns and a multi with a high sharpe.I definitely need to explore a bit further and hopefully learn to set up the scoring algorithm properly.Perhaps my simple method is what cause there to be many instances where the strategies selected appear to be all of the same type and likely very correlated.

Nevertheless, there's no current way to actually verify just how diverse the strategies are, so your idea does make sense as a way to provide that verification. Thinking about it from a developers perspective, it's unfortunately a pretty big job. The display of the matrix would be quite simple, but we'd need to find some way to store the equity lines for each of the individual strategies. Numbers for the individual strategies within a multi-system aren't currently stored, or even calculated, so we're a ways away from having such data even available.


Thats pretty much what I had in mind,but as you probably know,i am basically clueless as to the size of the programming job necessary.I am just an "idea" guy....

In your opinion,do you think the OC tends to collect highly correlated strategies if one runs thru all the rules?? Or would you say there is a fairly equal distribution of trending strategies combined with oscillating types...Utilising alternate data/periods,MC walk fowards is a great safeguard,but does it really help us in the selection process?

I guess the 2 formulas we currently have available are the cor() and alpha() formulas. These can both already be used to measure correlations with existing indexes (like the S&P 500 or DJIA). But they can't currently be used to measure a correlation against the equity line of an existing system.
I'll have to think about this a bit more. Perhaps if there were some way to export the equity line of an existing system into an Index, that Index could then be used within a cor() formula in a future strategy. If the cor() and alpha() and beta() numbers were then placed in the Summary tab or Performance Report, you'd at least be able to examine this manually, one strategy at a time. And the approach probably wouldn't be that hard to implement.


I think there would be value in having the ability to examine the stats manually,especially in the building of multis.Think about this.

You have 2 breakout systems that are highly correlated with each returning 25% APR,and you "MUST" add one more system. Would you chose another breakout system with returns of 25% and a correlation of 1,or would you prefer to add a system with a return of 12% but with a -1 correltion to the other two systems....

I am curious how OC approach would differ from a manual approach based on correlation..Think we would end up in the same place??

Allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Mon Jun 04, 2007 12:46 pm

In your opinion,do you think the OC tends to collect highly correlated strategies if one runs thru all the rules?? Or would you say there is a fairly equal distribution of trending strategies combined with oscillating types...Utilising alternate data/periods,MC walk fowards is a great safeguard,but does it really help us in the selection process?

As I said, there's a certain amount of this kind of filtering built-in indirectly already. If a new strategy is exactly correlated with an existing strategy, then there would be no increase in the multi-system's running score. So the new strategy wouldn't be accepted. But if it turns out that the new strategy decreases drawdowns, helps overall returns, and provides additional consistency over time, then the multi-system's score will be improved and the new strategy will be added.

It might be the case that that's what a negative correlation will provide: help in the areas where the existing system is weak. In particular, that generally means the drawdown areas. If a new strategy can help straighten the equity line, then it will generate a higher score and therefore be kept. So I think this indirect, built-in method for looking for significantly different strategies should work to a degree already, at least in theory.

I am curious how OC approach would differ from a manual approach based on correlation..Think we would end up in the same place??

I'm not sure, but it's a good question. It's true that you may end up with a couple highly correlated strategies right now. That isn't to say that they're not effective strategies, but only that they're not very diverse. It may come down to the question of whether you want non-correlated strategies or whether you want a higher score. In some cases, you may not get both. And it's these cases where I think your idea would help select diversity over returns.

I'll have to think about this a bit more, just to see if there's possibly a way to experiment with the idea before doing a full implementation. I've mentioned before that we often think the next big thing is right around the corner, only to find it didn't pan out as expected after implementation. It would be nice to experiment with this idea just to see how diverse the StrataSearch strategies already are before barreling forward.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm


Return to Curve-Fitting and Other Pitfalls

cron