out of sample test

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

out of sample test

Postby lucciri » Thu Sep 27, 2007 3:37 am

Please can someone help me, step by step to do these things
I want to:

1) create a strategy using 30 stocks (I knwow how to create a sector with 30 stocks)
2) use oneclick with date from 1997 to 2003
3) use the best scored strategy found out of sample (from 2003 to today)
4) show the out of sample equity line (from 2003 to today)

I wasn't able to do these things
lucciri
 
Posts: 3
Joined: Wed Sep 26, 2007 6:35 pm

Postby Overload » Thu Sep 27, 2007 11:23 am

The first step is to create a sector with your 30 stocks in it. You can do this from the Setups > Sector Setups menu.

Once that is complete, you will want to create a new OneClick Setup. Enter the Setups > OneClick Setups and highlight the "Small Sector Search" setup and then click "Copy". After giving your setup a name, you can then follow these steps:

1) create a strategy using 30 stocks (I knwow how to
create a sector with 30 stocks)

Navigate to the Sectors tab of the OneClick Setups. Select "Selected Sectors", and then click the "Uncheck All" button. Finally, double click your new 30-stocks sector to select it.

2) use oneclick with date from 1997 to 2003

Navigate back to the General tab. For the Evaluation Period Override, set the dates for 1997 to 2003.

3) use the best scored strategy found out of sample
(from 2003 to today)

Also on the General tab, check #1 in the Alternate Data Analysis. You can then set the period from 2003 to today and also select your 30-stock sector.

You'll then need to navigate to the Strategy Performance tab, where you can set up filters and scoring that include your Alternate Data evaluation. For more information on this, open the Help > Contents menu, and view the Alternate Data Analysis page within the "Advanced Features" chapter.

4) show the out of sample equity line (from 2003 to
today)

The Detailed Analysis will run against your main Evaluation Period by default. However, you can detach the multi-system from the OneClick Search and run it against whatever time frames you would like. You can do this by:

1) Right-clicking on the result in the Combination Results Listing and selecting Create Multi-System. Give the Multi-System a Name and click OK.

2) Open the Setups > Trade Settings menu and set your Evaluation Period on the General tab.

3) Use the Run > Run Combinations menu to run your new Multi-System.

4) When the run is complete, use the Display > Combination Results to display the result.

5) Right-click on the result in the Combination Results and select "Run Detailed Analysis". When complete, you can then view the equity line from 2003 to present.

Hopefully this gets you pointed in the right direction, but feel free to let us know if you have additional questions.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby lucciri » Sat Sep 29, 2007 5:00 am

thanks,

alternate data is useful , but I want that my oneclick search doesn't take into account 2003-today , I want 2003-today equity line to be pure out of sample. I understant that by checking #1 in the Alternate Data Analysis oneclick search uses 2003 today data for scoring, for me this is curve fitting.
In the help file I read "You would like your system to have minimal drawdowns in bear markets. For example, your primary search could be based on 2004-2006, but you also configure a filter to reject any strategies that have drawdowns greater than 35% from 2000-2002"
If a primary search is based on 1997-2003 and the system rejects strategies that does bad on 2003-today , the whole system is curve fitted

thank your
lucciri
 
Posts: 3
Joined: Wed Sep 26, 2007 6:35 pm

Postby Overload » Sat Sep 29, 2007 9:06 am

Because StrataSearch does not "train" systems according to specific time periods like many optimization software packages do, there is really no difference between how StrataSearch treats in-sample versus out-of-sample. It runs each period just once, first running and ensuring the "in sample" period passes all filters, and next running and ensuring the "out of sample" period passes all filters. But again, StrataSearch does not treat the two periods any differently.

There has actually been great debate regarding the benefits of "out of sample" testing, with some users insisting that the quality of a system is much greater if it can pass an "out of sample" test after first passing an "in sample" test. Other users, however, claim that because there is no training involved, the only benefits from the "out of sample" testing is that it merely adds more data to the evaluation.

You have indeed gotten to the heart of the issue quite nicely. If one adds an additional layer of automation such that the "out of sample" tests are performed automatically on every system, then doesn't the "out of sample" then become "in sample"? On the other hand, does it really make sense to do that "out of sample" testing manually when automation can do it so much more efficiently?

In StrataSearch, we refer to it as "Alternate Data" rather than "Out of Sample" data for this reason. Ultimately the choice is yours how you wish to use it. If you would prefer not to use the integrated Alternate Data tests, but instead run the numbers on the alternate period manually afterwards, that is up to you. Or you can include the Alternate Data tests in your automated searches as described. But whether you do your filtering on the data manually or automatically, your end results will be exactly the same. In both cases, you'll be selecting only the systems that worked in the alternate data period.

Also... I would disagree with the assumption that the whole system is curve fitted when automation is added to the Alternate Data testing. Adding a variety of tests against both alternate time periods and alternate sectors can greatly reduce the odds of curve fitting. As well, StrataSearch has a very large number of tests to avoid curve fitting, of which Alternate Data testing is only one. Parameter Shifts, Monte Carlo simulations, and many other performance values can help make sure your systems are not curve fitted. See the "Curve-Fit Protection" tab of the OneClick Setups, and click the Help button while viewing that tab for more information.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby lucciri » Sat Sep 29, 2007 9:53 am

Pete,,

"On the other hand, does it really make sense to do that "out of sample" testing manually when automation can do it so much more efficiently?"

Ok now is clear, thank you.

Please don't misunderstand I like SS a lot, I just don't want to underestimate its capabilities

"SS does not "train" systems according to specific time periods like many optimization software packages do"

Please can you explain? I used Enhaced System Tester in Metastock, SS is many times more powerful and flexible but what is the difference conceptually?
i.e Metastock's optimizer finds out the best moving average (or RSI,MACD , ect) for the input time periods (in sample)..... the trading rule that worked best during the input time period
SS's more powerful optimizer finds out the best combinations of trading rules for the inputs (in sample)....... the "linked" trading rules that worked best during the input time period

thanks
lucciri
 
Posts: 3
Joined: Wed Sep 26, 2007 6:35 pm

Postby Overload » Sat Sep 29, 2007 2:19 pm

That's a good question, and I'm going to respond in depth because I think it's worth discussing. One of the primary differences is that the StrataSearch automation finds trading systems from scratch, while optimization software normally finds the best parameter sets within an existing system. While there is a lot of overlap between the two methods, there are significant differences as well.

Within optimization software, it is their optimization algorithms that are considered the most important. Normally, these algorithms are proprietary and often hidden from you. And, in short, they are the method used to identify the best parameter set. As an example, suppose you are using optimization software to identify the best RSI() parameter set between 5 and 20. Optimization software might go through these methods:

PHASE 1:
RSI( 5 ) = 14% APR
RSI( 10 ) = 15% APR
RSI( 15 ) = 19% APR
RSI( 20 ) = 17% APR

PHASE 2:
RSI( 17 ) = 20% APR
RSI( 19 ) = 17% APR

PHASE 3:
RSI( 18 ) = 21% APR (winner)

This is a very simplified example, since optimization software is usually robust enough to consider more performance criteria than just the APR. Nevertheless, it shows how the software "efficiently" drills down into the winning parameter set without having to test every RSI value between 5 and 20. Normally, within the tests of each RSI value, an "out of sample" test is included to verify the quality of each value.

As you can see, however, there are many RSI values that are never tested at all. For example, the values 11-14 as well as 16 are never considered. By using optimization software that has algorithms like this, you need to trust that they are making the best decisions.

By contrast, StrataSearch will test all values between 5 and 20, assuming you let the search continue long enough to do so. There is no parameter set that cannot be reached and tested. There are several reasons why we've chosen to do things this way:

1) StrataSearch runs analyses 50-100 times faster than most other software packages, so we can afford to run every possibility.

2) Rather than basing the quality of your trading systems on our own internal algorithms, we believe the quality should be based on the results themselves. Let every parameter set be judged on its own, and not by our own internal algorithms.

3) StrataSearch already has an unprecedented level of automation, and users may already be distrustful of the results. If our own algorithms made decisions for you, you would need to trust that StrataSearch was making good decisions internally. However, as it is now, the search and selection criteria are entirely in your hands through the setups. There are no decisions that are made that you cannot control and change within the configurations.

In the end, the best way to describe it is that each system in StrataSearch is judged independently, without regard to prior iterations. No parameter set is unreachable, and no decisions are made for you. With optimization software, you are paying for their internal algorithms, and you must trust that they are making good decisions for you.

To be fair, the best optimization software packages let you customize the optimization by ranking a dozen or more performance values. And some optimization software packages are quite good. But few, if any, allow you to filter and rank with the hundreds of performance values that are available to you in StrataSearch.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm


Return to Curve-Fitting and Other Pitfalls

cron