WFA

Let us know your likes and dislikes, or what you'd like to see next. We listen!

Re: WFA

Postby taowave » Sun Mar 18, 2012 6:57 pm

Hi Pte,
Thanks for the input.The first go round I did not enable optimisation,but was curious to see the effect.
Very interesting that you have as many as 5-15 variables.I dont know if I could "emotionally" trade like that as I am predominantly a discretionary trader(patterns) who places equal emphasis on fundamentals.

As most patterens come down to reversal vs trending/breakout,I employ SS to be my research assistant and come up with as much un-biased research as possible.
Intuitively,having an exhaustive approach appears to be absolute,which I like as my discretionary approach is anything but absolute.With that said,I do see your point,regarding systems with many variables.

Is there any loose guidline for what percent the est sampling per period should be to get a near exhaustive approach.I dont mind ballpark returns,but having a system go from -8% at 123% to + 5% at 309% is too big a variance.

If I may ask,do you find that systems with 10-15 variables actually hold up OOS?

Do you always walk foward?


Allan



Overload wrote:Thanks, I got your WFA and was able to look at what you're doing a little closer. And I have several thoughts:

1) First, you may find it difficult to match the results of a WFA from one program to another. It is not that the same parameters sets will not be run, but rather that there are many settings regarding how WFAs are handled at each interval and you would need to be very sure both programs are handling everything identically. If even one setting is different, you will be comparing apples to oranges. This includes things like how the "best" parameter set is chosen, the number of days in the interval, the number of days in the evaluation period, whether positions are exited at the end of the interval, etc. Even further, you would need to be sure the results themselves are identical, including use of portfolio, removing best/worst trades, fixed versus variable equity evaluations, spread and slippage, etc. This isn't to say a WFA can't be compared from one program to another. But doing so is a significantly more complicated task that simply comparing a static system from one platform to another.

2) I noticed in your WFA that you have both phases of optimization turned on. This is incorrect based on what you are trying to do. The first phase is basically redundant, since there is no need to optimize variables when you only have one variable. But the second phase is most definitely problematic since you are potentially inserting new trading rules into your strategies at each interval. I don't think that's what you want to do. So you should turn off Optimization altogether.

3) If your WFA is truly exhaustive, you should be able to run it countless times and get the exact same result every time. Since that didn't apparently happen with you, the only answer is that you weren't letting it run long enough at each interval. StrataSearch does not sequentially test each parameter set, but instead uses its common approach of testing random combinations for a specified time limit. While I agree this is less efficient if you only have one or two variables, I've found it is pretty rare in StrataSearch that you only have one or two variables. Most systems I've found have 5 or 10 or even 15, and it would take a lifetime to test every possible combination. In this case, the random sampling becomes very valuable, but the time you spend on each interval becomes a critical factor. In my opinion, it is as important as any one of the variables.

To better explain my technique of partial sampling, consider both the Monte Carlo Simulations and the Parameter Shift Testing. Both of these techniques are methods of tweaking the algorithms slightly with the intention of discovering how robust the trading system actually is. A partial sampling in a WFA is a similar concept. If you can only get the 2nd best, or 3rd best parameter set in some of your intervals, would your system still be profitable? If not, then your system may not be very robust and could fail moving forward. Again, it is similar to a Parameter Shift Test. If you are using a moving average of 19 days instead of 20, does your system fall apart? If so, that system lacks robustness and could very well fail moving forward.

Pete
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby Overload » Mon Mar 19, 2012 11:42 am

Actually 5 to 15 variables in a trading system is pretty common. Remember this is variables, not trading rules. In fact, if you run any standard AutoSearch or OneClick in StrataSearch, then right-click on the result and select Create Optimization Strategy, you'll see there are often 5 to 15 variables. Less than 5 is probably more the exception.

As for your own experience with it going from -8% to +5%, you would need to look more closely at the results to determine why that was the case. Did you simply miss out on one trade that had a 150% gain? Or was it spread out evenly across all the trades?

In any case, my suggestion about the limited sampling remains the same. If your return is +5% at 100% sampling, but -8% at 95% sampling, I would consider that a very big red flag. Either your returns are based on a couple high flyers like I suggested above, or the system is not very robust and could fall apart with just a small variance from the norm. In any case, that comparison of limited sampling versus complete sampling has raised the flag and made you aware that you need to look closer. I believe that is a good thing.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Re: WFA

Postby taowave » Mon Mar 19, 2012 8:46 pm

Hi Pete,
Now that I fully understand what you are saying,it does sound like a very good thing.I guess my next naive question is how do I know I sampled 100% vs 95% as opposed to 70%?
Wouldnt that be useful? The only "gauge" I see is the estimated sampling period,and I am not sure what one discerns from 123% vs 306%.

As for variables,I have always intentionally kept them less than 3.Thats very interesting that you have 5-15.
I personally find that when I have a large number of variables,I often do not have an intuitive understanding of the system produced..

If you find a system that performs well according to your criteria,do you simply trade it regardless if you have a good "feel" for it??
Or as a system trader,you strictly go by the numbers?
Appreciate the help,

Allan



Overload wrote:Actually 5 to 15 variables in a trading system is pretty common. Remember this is variables, not trading rules. In fact, if you run any standard AutoSearch or OneClick in StrataSearch, then right-click on the result and select Create Optimization Strategy, you'll see there are often 5 to 15 variables. Less than 5 is probably more the exception.

As for your own experience with it going from -8% to +5%, you would need to look more closely at the results to determine why that was the case. Did you simply miss out on one trade that had a 150% gain? Or was it spread out evenly across all the trades?

In any case, my suggestion about the limited sampling remains the same. If your return is +5% at 100% sampling, but -8% at 95% sampling, I would consider that a very big red flag. Either your returns are based on a couple high flyers like I suggested above, or the system is not very robust and could fall apart with just a small variance from the norm. In any case, that comparison of limited sampling versus complete sampling has raised the flag and made you aware that you need to look closer. I believe that is a good thing.

Pete
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby taowave » Tue Mar 20, 2012 9:52 am

Well,I have a request!!

I would like a filter indicating what percent of the total number of possible optimizable combinations has been selected:)

Thanks,

Allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby Overload » Tue Mar 20, 2012 11:47 am

You're correct that there isn't currently a way to exactly identify the percentage of sampling you're using. For now, you need to run your own tests and create your own estimates. But I will put this on the list as a possible future enhancement.

For me personally, everything I do is by the numbers, although it's true that identifying which numbers to use is a personal choice.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Re: WFA

Postby taowave » Tue Mar 20, 2012 5:45 pm

So,if I understand everything,SS has an added advantage of having the ability to display the "robustness" of the WFA by inputting low
estimated sampling periods that are not exhaustive.If the returns from that simulation are satisfactory,that would indicate that those returns are the minimum we could expect as we increase the ESP.Is that correct?

Do you have any thoughts on OOS vs IS and the ratio of the two?
Is that a useful measure?
Allan

Overload wrote:You're correct that there isn't currently a way to exactly identify the percentage of sampling you're using. For now, you need to run your own tests and create your own estimates. But I will put this on the list as a possible future enhancement.

For me personally, everything I do is by the numbers, although it's true that identifying which numbers to use is a personal choice.

Pete
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby Overload » Wed Mar 21, 2012 10:30 am

So,if I understand everything,SS has an added advantage of having the ability to display the "robustness" of the WFA by inputting low
estimated sampling periods that are not exhaustive.

Correct.

If the returns from that simulation are satisfactory,that would indicate that those returns are the minimum we could expect as we increase the ESP.Is that correct?

Not quite. The returns from a partial sample give you one possible outcome. That outcome is not necessarily the minimum that can be expected.

Do you have any thoughts on OOS vs IS and the ratio of the two?
Is that a useful measure?

Sorry, I've never used such a ratio, but it's an interesting idea.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Re: WFA

Postby taowave » Wed Mar 21, 2012 12:40 pm

Hi Pete,

Perhaps I misphrased the partial sample question.As you explained,the WFA may not be exhaustive.Am I correct that in a partial sample WFA,SS selects the best IS results and than walks foward creating OOS results?

Assuming that is correct,the WFA OOS results produced in a partial sample will be the ones associated with the highest IS simualtion.So a simulation with partial sampling should be the floor/minimum for IS testing(not OOS),and as we approach an exhaustive WFA the IS returns used would have to be >= to the partial sample.

As I believe you have pointed out,while an exhaustive search will produce an IS result >= a partial sample,that does not necessarily equate to a higher OOS simulation.In fact,you could run an exhaustive WFA,and while the IS results would be the greatest,the OOS could be lower than in a partial sampling.Simply put,max IS returns do not necessarily equate to positive OOS returns.

I think I now have a much better grasp on WFA,and in particular its shortcomings.I will say that SS has the best set of evaluation tools that I have seen to break down the walk foward.

Thanks,

Allan

If the returns from that simulation are satisfactory,that would indicate that those returns are the minimum we could expect as we increase the ESP.Is that correct?

Not quite. The returns from a partial sample give you one possible outcome. That outcome is not necessarily the minimum that can be expected.



Pete[/quote]
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby Overload » Wed Mar 21, 2012 1:12 pm

Again that's close, but some clarification is still needed I think.

Am I correct that in a partial sample WFA,SS selects the best IS results and than walks foward creating OOS results?

Maybe. It's worth clarifying that it selects the best IS results of the ones it has tested within that sampling. The best parameter set may or may not be included in that partial sample.

Assuming that is correct,the WFA OOS results produced in a partial sample will be the ones associated with the highest IS simualtion.

Yes, but again only if you mean the parameter sets that were actually selected within the partial sample.

So a simulation with partial sampling should be the floor/minimum for IS testing(not OOS),and as we approach an exhaustive WFA the IS returns used would have to be >= to the partial sample.

No, this is where you lost me. Assuming we have a partial sampling of 95%, that means 95% of the possible combinations will be tested and potentially included in selection for that interval. That also means that 5% will not even be considered regardless of whether they perform better or worse than the combination that ultimately gets selected. It sounds like you might be assuming that a 95% partial sample means the best 95%. That is not the case. It is a random 95%.

Because there is no way of knowing whether those 5% would have performed better or worse, it is impossible to say whether the partial sample performs better or worse than the complete sample. In fact, you could run a partial sample WFA many times and have it come up with a different results every time. The 5% of the combinations that are left out will change randomly at each interval, and that is that shift of selection that creates the benefit of this partial sample test.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Re: WFA

Postby taowave » Wed Mar 21, 2012 3:09 pm

Hi Pete,
Yes,the best IS that SS has selected in the partial testing.
Maybe. It's worth clarifying that it selects the best IS results of the ones it has tested within that sampling. The best parameter set may or may not be included in that partial sample.


Agreed,only the parameter sets that were selected
Yes, but again only if you mean the parameter sets that were actually selected within the partial sample.




What I am trying to say is that if you perform a 95% sampling,the IS parameters selected that generated the most profit will be the minimum IS profitability. The "remaining" 5% IS sampling can only select parameters that exceed the profitability of the 95% sampling.Thus,the IS results will be at least what the initial 95% sampling tested produced.Thats why i naively called it the minimum.Perhaps floor would have been a better description.Please note,i am only referring to IS,not OOS.If I am not mistaken,SS doesnt break down IS vs OOS,so this is really a very long winded moot point:)
No, this is where you lost me. Assuming we have a partial sampling of 95%, that means 95% of the possible combinations will be tested and potentially included in selection for that interval. That also means that 5% will not even be considered regardless of whether they perform better or worse than the combination that ultimately gets selected. It sounds like you might be assuming that a 95% partial sample means the best 95%. That is not the case. It is a random 95%.




I now agree 100%.The best OOS result could come at any interval,and the best set of IS parameters may not produce the best OOS.
What I do like about an exhaustive search is at least I can walk away saying "If one selects the best IS parameters,these are the OOS results.
It seems a bit easier to accurately benchmark that way.
Because there is no way of knowing whether those 5% would have performed better or worse, it is impossible to say whether the partial sample performs better or worse than the complete sample. In fact, you could run a partial sample WFA many times and have it come up with a different results every time. The 5% of the combinations that are left out will change randomly at each interval, and that is that shift of selection that creates the benefit of this partial sample test.

Pete[/quote]
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby taowave » Fri Mar 23, 2012 1:42 pm

Hi Pete,
I am following your lead and running a WFA with 6 optimizable parameters.Regardless of what search times i put in,the est sampling per period remains at <1%.

Any thoughts/advice?

Anyone else ?
Allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: WFA

Postby Overload » Fri Mar 23, 2012 3:14 pm

The thought is that with a good, static set of trading rules, you won't need to do a large sampling to create a worthwhile system. So maybe you'll get a good parameter set once every 1,000 combinations as an example. In some ways it's similar to running an AutoSearch. In an AutoSearch, there might be trillions of possible combinations, but your APR isn't going to improve significantly after about 500,000, for example.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Previous

Return to Comments and Suggestions