Parameter Shift within Multi System

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

Parameter Shift within Multi System

Postby damcuad » Sun May 22, 2016 4:02 pm

Hi

I have a few questions regarding parameter shifts and curve fitting I'm hoping I can get help with. Let me preface by saying I am trying to set up a multisystem search for a single symbol.

1. When I set up the Parameter Shift test (using variable equity terms instead of fixed), I am noticing that when I run a strategy from the multisystem separately using the same dates and prices that were used for the search, I'm not seeing bars which correspond to my cutoffs. 50% for 1S. 25% for 2S, 12.5 for 3S. For example, one of the 2S bars will be less than 25% of the 0 bar (I have the Up and Down Required box checked in Trade Settings). This is especially common the higher I go in Strategy #. I'm sure there is some "silly" reason for this, but I'm not seeing it.

2. I understand that due to the "arranging" of strategies within a multisystem that the Parameter shifts will experience a drop off from what I see from each strategy, but why is it so drastic? For instance, the search result has 75 trades (70 win 5 loss) and the Performance tab shows say 67 trades (63 win 4 loss) for the multisystem and yet the parameter shift chart for the multisystem is drastically worse.

3. Should I be preoccupied with parameter shifts if I am using 24 years worth of data? I would think that it wouldn't be as much of an issue as if I were using 5 years. How many years of data (10, 15, 20 etc) should one be really concerned with robustness IF at 24 years one does not need to be THAT concerned?

4. Is preoccupation with seeing a nice Avg Ann Return Parameter Shift chart the best way to insure robustness?

5. If a multisystem search reaches the goals set for the only two scoring factors in the Multisystem tab, will continuing the search seek to improve on the strategies "score" in the Strategy tab? Optimizing towards a higher percentage of the Parameter Shift tests for instance?

6. Ultimately, I would like the Multisystem search to "push" towards both Avg Annual Return and Minimizing changes in Parameter shifts EQUALLY. Is it possible to "push" for a nicer Parameter Shift chart via the Multisystem tab? If not, what settings would you recommend to have the search "push" toward higher returns and robustness equally? I understand that I will be sacrificing Avg Ann Return for Robustness, but I would like to better control the amount each is "pushed" for by the search. If you recommend other techniques for guarding against curve fitting, such as Sharpe, please provide examples of values to consider. I've seen in the manual where it states that a "high number" is preferred for the Sharpe ratio, but without a range, the term "high number" doesn't really help.

I know there is a lot here. I really appreciate any guidance/insight offered.

Thank you
damcuad
 
Posts: 9
Joined: Fri Dec 04, 2015 1:00 pm

Re: Parameter Shift within Multi System

Postby Overload » Sun May 22, 2016 5:21 pm

1. When I set up the Parameter Shift test (using variable equity terms instead of fixed), I am noticing that when I run a strategy from the multisystem separately using the same dates and prices that were used for the search, I'm not seeing bars which correspond to my cutoffs. 50% for 1S. 25% for 2S, 12.5 for 3S. For example, one of the 2S bars will be less than 25% of the 0 bar (I have the Up and Down Required box checked in Trade Settings). This is especially common the higher I go in Strategy #. I'm sure there is some "silly" reason for this, but I'm not seeing it.

Parameter Shift results are based on an average of the up and down, not both up and down individually. That could explain part of it.

2. I understand that due to the "arranging" of strategies within a multisystem that the Parameter shifts will experience a drop off from what I see from each strategy, but why is it so drastic? For instance, the search result has 75 trades (70 win 5 loss) and the Performance tab shows say 67 trades (63 win 4 loss) for the multisystem and yet the parameter shift chart for the multisystem is drastically worse.

I don't know enough about your multisystem setup, but a multi-system can be very different than an individual strategy. Also, if the parameter shift run is significantly worse, that is a sign of curve-fitting.

3. Should I be preoccupied with parameter shifts if I am using 24 years worth of data? I would think that it wouldn't be as much of an issue as if I were using 5 years. How many years of data (10, 15, 20 etc) should one be really concerned with robustness IF at 24 years one does not need to be THAT concerned?

Parameter shifts are different than simply more data. It is testing alternate parameters within the formulas. But it's up to each trader to determine how much value they place in each test.

4. Is preoccupation with seeing a nice Avg Ann Return Parameter Shift chart the best way to insure robustness?

Average return isn't a good test of robustness in itself, but it's not bad to still use it.

5. If a multisystem search reaches the goals set for the only two scoring factors in the Multisystem tab, will continuing the search seek to improve on the strategies "score" in the Strategy tab? Optimizing towards a higher percentage of the Parameter Shift tests for instance?

If the score is already 100%, then it won't be able to find any additional improvements to the multi-system. But it does not automatically stop looking.

6. Ultimately, I would like the Multisystem search to "push" towards both Avg Annual Return and Minimizing changes in Parameter shifts EQUALLY. Is it possible to "push" for a nicer Parameter Shift chart via the Multisystem tab? If not, what settings would you recommend to have the search "push" toward higher returns and robustness equally? I understand that I will be sacrificing Avg Ann Return for Robustness, but I would like to better control the amount each is "pushed" for by the search. If you recommend other techniques for guarding against curve fitting, such as Sharpe, please provide examples of values to consider. I've seen in the manual where it states that a "high number" is preferred for the Sharpe ratio, but without a range, the term "high number" doesn't really help.

You can weight them equally in the search, but you'll still need to give them a range that is comparable for each search criteria. And that's not a perfect science. It might take some experimentation to get that working the way you want.

Pete
Overload
 
Posts: 2225
Joined: Wed Nov 30, 2005 12:14 pm

Re: Parameter Shift within Multi System

Postby damcuad » Mon May 23, 2016 9:14 am

Pete,

Thanks for getting back to me so quickly

"Parameter Shift results are based on an average of the up and down, not both up and down individually. That could explain part of it."
I think that is part of it. I thought that when I made the cutoff 50% for a 1S shift it would do so on both sides. I'm noticing I'm getting nice bars to the right, but so much towards the left.

"Parameter shifts are different than simply more data. It is testing alternate parameters within the formulas. But it's up to each trader to determine how much value they place in each test."
What I was trying to get across with the years of data is basically I feel that a curve fitted system for 24 years of data is more robust than a curve fitted system for 5 years of data. In other words if this particular system worked for 24 years, then it has a higher probability of not failing in the near future than a system which was successful for 5 years. Or is this line of thinking completely off?

What techniques/search parameters would you recommend to avoid curve fitting within a single symbol search? Can you recommend some values to start off with as well?

Thank again
Damian
damcuad
 
Posts: 9
Joined: Fri Dec 04, 2015 1:00 pm


Return to Curve-Fitting and Other Pitfalls

cron