Curve Fitting Versus Statistical Robustness

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

Curve Fitting Versus Statistical Robustness

Postby JLG » Thu Nov 01, 2007 2:20 pm

My current objectives focus on maximum dollar gain with acceptable drawdown and less than tolerable consecutive losing trades and not much else. I am not currently rejecting strategies for any other reason.

I tend to make three assumptions:

Markets are skewed and have fat tails; any system needs to be structured to deal with this.
Markets have periods where you are better off to be in out out entirely. (This is supported by results that move quickly in general from minimum to maximum postions and vice versa).
Statistical robustness is key. So I tend to look at longer time periods -- 15 years of US history and systems that have higher numbers of trades to increase the statisitcal significance on trades and coverage of market characteristics.

This brings me to the Strataserach curve fitting question.

To deal with curve fitting I tend to use: alternate data sets, Monte Carlo, numbers of trades and parameter shift analysis, K Ratio and Z score.
But I tend to avoid use of Skew, Kurtosis, active periods, days held, number of winning periods, standard deviations, bests/worsts trades/periods.
My concern is this may leave me quite vulnerable to curve fitting.

Any suggestions of what I could or should do differently?
JLG
JLG
 
Posts: 100
Joined: Thu Oct 25, 2007 3:32 pm

Re: Curve Fitting Versus Statistical Robustness

Postby Dacamic » Thu Nov 01, 2007 2:46 pm

Jan Gullett wrote:To deal with curve fitting I tend to use: alternate data sets, Monte Carlo, numbers of trades and parameter shift analysis, K Ratio and Z score. But I tend to avoid use of Skew, Kurtosis, active periods, days held, number of winning periods, standard deviations, bests/worsts trades/periods. My concern is this may leave me quite vulnerable to curve fitting.

Any suggestions of what I could or should do differently?

My three primary tests for robustness are:
    1. Parameter stability, i.e., parameter shift analysis;
    2. Performance consistency throughout evaluation period (evaluated visually); and,
    3. Monte Carlo analysis.
For what it's worth, I don't use any form of optimization, and so include all data in my evaluation period rather than dividing it into "in-sample" and "out-of-sample" batches.
Steve
Dacamic
 
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm

Postby Overload » Fri Nov 02, 2007 8:25 am

Statistical robustness is key. So I tend to look at longer time periods -- 15 years of US history and systems that have higher numbers of trades to increase the statisitcal significance on trades and coverage of market characteristics.

I agree that statistical robustness is key. But I think those large numbers of trades can come from 2 places: a longer evaluation period or a larger sector. While including the boom/bust years can be beneficial, high returns will often be skewed toward those periods. So one has to be careful that there's still significant benefit to trading the system outside of those highly volatile periods.

Using a larger sector, on the other hand, provides the benefit of large numbers of trades while allowing you to focus on market conditions similar to the current market. Obviously there are pros and cons either way. But it's worth considering.

To deal with curve fitting I tend to use: alternate data sets, Monte Carlo, numbers of trades and parameter shift analysis, K Ratio and Z score.
But I tend to avoid use of Skew, Kurtosis, active periods, days held, number of winning periods, standard deviations, bests/worsts trades/periods.
My concern is this may leave me quite vulnerable to curve fitting.


I think one of the most helpful tools for avoiding curve-fitting is consistency of the system, both over time and across trades. While you're including the K-Ratio, I think there is the opportunity for more. Evaluation of monthly or yearly standard deviations probably would not hurt. But checking kurtosis could be helpful for verifying consistency of your trades. An alternative that I've had some pretty good success with is to evaluate the "Select Net Profit" compared to the "Net Profit". Select Net Profit is simply the Net Profit but excludes any gain/loss from trades that are more than 3 deviations from the average. So it gives a good indication of how the system would normally perform without the few, infrequent skewed trades. If there is a large difference between the Select Net Profit and the Net Profit, this is not a system I would normally trade.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Postby JLG » Fri Nov 02, 2007 12:58 pm

Thanks for your comments. I guess what I am seeking probably doesn't really have an answer in some respects.

I guess the curve fitting problem is really a matter that the test period may not be a representative sample of the forward period. Historical systems which are developed to profit from the tails rely on a small, rare tail event sample size and that may not recur in the future. So any real test of robustness will need to penalize or elminate the benefit of trades from the fat tails focusing on the more probable trades, otherwise the profit from these rare events may skew results on the more frequent, but much smaller opportunities.

Someone wise once told me "follow the money" but I guess in the case of tail event profit this is dificult, due to the limitations of the statistics. Unfortunately this can eliminate a major source of profit.
JLG
JLG
 
Posts: 100
Joined: Thu Oct 25, 2007 3:32 pm

Postby Overload » Fri Nov 02, 2007 1:55 pm

I guess the curve fitting problem is really a matter that the test period may not be a representative sample of the forward period.

There has been much discussion about this, but I'd prefer to say that such a system simply lacks robustness. There are many, many ways to test how robust a system is (ranging from parameter shifts to alternate data analyses). But the question is really how robust do we want our system to be? In my experience the more robust a system is, the less it can take advantage of the risk/reward opportunities of a given time period. Thus, it becomes a personal decision regarding how to adjust that balance.

It's rather ironic that people pursue system trading to avoid having to make personal trading decisions, and yet system trading still requires its own share of personal trading decisions.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Postby JLG » Fri Nov 02, 2007 2:40 pm

Pete,

Lack of robustness is probably a better description. We just do not know, nor can we, whether the test period is a representative sample of the forward period on high impact fat tail events, due to the small sample size and unique nature of these events (but it would be worth a lot if we could). More frequent events are more likely to be represented reliably in the test period though shifts in the nature of the market could still prevail.

Your ironic observation reinforces a notion I have had that systems can reduce risk in a fairly straightforward manner while increasing gains in a reliable manner is a greater challenge. I guess that is why many choose make money a nickel at a time and then increase those returns with leverage whether it be Long Term Capital Managment or Renaissance, not to mention the folks who regularly sell out of the money puts and calls in exchange for taking the fat tail risk. This seems to work very nicely until a fat tail comes along...
JLG
JLG
 
Posts: 100
Joined: Thu Oct 25, 2007 3:32 pm

Postby Dacamic » Mon Nov 05, 2007 4:47 pm

Jan Gullett wrote:Someone wise once told me "follow the money" but I guess in the case of tail event profit this is dificult, due to the limitations of the statistics. Unfortunately this can eliminate a major source of profit.

In many profitable systems, each entry signal is a "tail" event because it was triggered by a low probability combination of variables. Cast across a large population -- created by time, sector size or both -- we might see many signals, tempting us to believe they are common; however, they are still rare regardless of absolute number. For example, my systems search across more than 10,000 symbols daily, most days not finding any defined set-ups and rarely more than ten. Even at the extreme, ten signals from 10,000+ chances reflects a very small percentage occurrence. From this perspective, we see that keying on unusually profitable trades requires us to find set-ups that define tails within tails. Even though "tail tails" tend to be fatter than many believe, developing a profitable system primarily focused on the far end of profit distribution curves is an understandably difficult task.

With the above being said, it's still reasonably possible to take advantage of tail trades by using systems that follow an old adage: cut your losses and let your profits run. Quite frankly, some of my systems depend upon outlier trades to be profitable. People might argue that is a sign of insufficient robustness, while I suggest it's staying in the game until a very unique profit opportunity appears. In other words, I usually prefer to toss a handful of darts at the board rather than trying to hit the bullseye with only one dart.
Steve
Dacamic
 
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm


Return to Curve-Fitting and Other Pitfalls

cron