Adjustments to Walk-Forward Analysis

Let us know your likes and dislikes, or what you'd like to see next. We listen!

Adjustments to Walk-Forward Analysis

Postby Kevin_in_GA » Sun Apr 22, 2012 7:46 pm

I just figured out how to use this - VERY cool and lots of value. You really might want to have some specific tutorials on this, since it is both powerful and not offered by many other vendors.

A quick suggestion - the evaluation period should have an option for weekly or monthly periods, rather than just days.
Kevin_in_GA
 
Posts: 171
Joined: Wed Aug 04, 2010 10:14 am

Re: Adjustments to Walk-Forward Analysis

Postby Overload » Mon Apr 23, 2012 9:19 am

A quick suggestion - the evaluation period should have an option for weekly or monthly periods, rather than just days.

Do you mean the "intervals"? Look at the Dynamic Strategies tab of the OneClick Setup. There you'll find a whole bunch of settings, including the New Strategy Frequency that lets you define whether you want the interval to be Daily, Weekly, Monthly, Yearly, or a custom number of days.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Re: Adjustments to Walk-Forward Analysis

Postby Kevin_in_GA » Mon Apr 23, 2012 9:33 am

Overload wrote:
A quick suggestion - the evaluation period should have an option for weekly or monthly periods, rather than just days.

Do you mean the "intervals"? Look at the Dynamic Strategies tab of the OneClick Setup. There you'll find a whole bunch of settings, including the New Strategy Frequency that lets you define whether you want the interval to be Daily, Weekly, Monthly, Yearly, or a custom number of days.

Pete


Right, but the rolling evaluation period is only in days. If I were to run a WFA on the weekly timeframe, I would want to have the option of having the evaluation period frequency in weeks as well.
Kevin_in_GA
 
Posts: 171
Joined: Wed Aug 04, 2010 10:14 am

Re: Adjustments to Walk-Forward Analysis

Postby Overload » Mon Apr 23, 2012 10:42 am

If you want 4 weeks, you can just enter 28 days.

I think you might not be understanding what the Rolling Evaluation Period is, and how it differs from the New Strategy Frequency.

The New Strategy Frequency identifies how frequently parameter sets will be adjusted. So, for example, if you have it set to Monthly, a search will be run on a monthly basis to identify which parameter set will be used in the next month.

The Rolling Evaluation Period identifies how much data will be used when determining which parameter set will be used. So if your New Strategy Frequency is set to Monthly, and the Rolling Evaluation Period is set to 365, you'll use the most recent year's data when determining which parameter set will be used each month.

I don't think it should make a difference whether you enter 4 weeks or 28 days. In either case, you'd be using 28 days worth of data to identify which parameter set will be used in the next interval.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Re: Adjustments to Walk-Forward Analysis

Postby Kevin_in_GA » Mon Apr 23, 2012 12:09 pm

So the rolling evaluation is based on Calendar Days and not trading days? If so that is a little easier to use.
Kevin_in_GA
 
Posts: 171
Joined: Wed Aug 04, 2010 10:14 am

Re: Adjustments to Walk-Forward Analysis

Postby Overload » Mon Apr 23, 2012 12:34 pm

Yes, it is based on Calendar Days.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Re: Adjustments to Walk-Forward Analysis

Postby Kevin_in_GA » Fri Apr 27, 2012 9:16 pm

There is inconsistency then in how SS uses "days" as input. For example, the calculation of PROC, RSI, MACD and a host of other indicators is based on TRADING days, while lookback periods are in CALENDAR days?

Would it be hard to revise the rolling evaluation period to using trading days, and also to allow weeks as a period input if the new strategy frequency is also weekly?
Kevin_in_GA
 
Posts: 171
Joined: Wed Aug 04, 2010 10:14 am

Re: Adjustments to Walk-Forward Analysis

Postby Overload » Sat Apr 28, 2012 10:10 am

You're correct that the two uses are different, but you're going to need to convince me that that's a problem. From what I can tell, the different implementations are appropriate for the two different uses.

The Rolling Evaluation Period in the Dynamic Strategies is really the identical setting of the Evaluation Period in the Setups > Trade Settings menu, or the Evaluation Period on the General tab of the OneClick Setup. Those happen to be defined with Start Date and End Date, but I'd suggest that those more closely resemble periods of Calendar Dates rather than periods of Trade Dates. The primary difference is that there is a specific Start Date that does not change from one symbol to another regardless of the number of price bars available for each symbol.

Suppose that on the Setups > Trade Settings menu, our Evaluation Period was defined like this:

End Date: 12/31/2011
Trading Days In Evaluation Period: 252

Using the above settings, we could force our Evaluation Period to use Trading Days instead of Calendar Days. But I'd suggest that this would not be a benefit. True, you could clearly define the exact number of days in your evaluation period, but the problem is that due to missing prices, some stocks may have different start dates than other stocks. Different symbols often have different numbers of price bars for the same period, which creates different start dates when you include data based on the number of bars.

The Rolling Evaluation Period defines how much data will be used in the Evaluation Period, just as the Start Date and End Date do. But because it is a walk-forward process with multiple intervals, it's impossible to specify the dates of each interval individually. So the next best thing is to specify the period in Calendar Days. The End Date is already known, but the Start Date can be identified by subtracting 365 days (for example) from the End Date to create an exact Start Date for that interval that applies to all symbols.

So, like I said, you'll need to convince me why we would want to use Trading Days instead of Calendar Days for the Evaluation Period. From what I'm seeing, Calendar Days is the appropriate choice there.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Re: Adjustments to Walk-Forward Analysis

Postby Kevin_in_GA » Sat Apr 28, 2012 4:59 pm

I see your point, but still think that the rolling evaluation period should be thought about in terms of "bars" rather than calendar days. If I look at performance of a stock I look at PROC (trading days), the same with evaluating strategies.

We can agree to disagree on this - I can see why you might leave this as is, and either way I appreciate the ability to do the WFA as it is a great way to really pressure-test a strategy.
Kevin_in_GA
 
Posts: 171
Joined: Wed Aug 04, 2010 10:14 am

Re: Adjustments to Walk-Forward Analysis

Postby Overload » Sat Apr 28, 2012 6:40 pm

Fair enough. I'll keep an open mind to the idea, and maybe I'll even come around at some point.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm


Return to Comments and Suggestions

cron