Alternate Data

Strategies might be attractive on the surface but fail moving forward. Why? Here we discuss the qualities of effective trading strategies, and the many traps one should avoid.

Alternate Data

Postby Schutten » Fri Oct 26, 2007 6:49 am

Hi ,

I have been browsing the forum quite alot. I bumped into some nice
discussions about out of sample data. I have got a phylosofical
question. If statasearch develops a system which succesfully passes
alternate period 1,2 and 3 with minimum setting 75%. If this is the
case would not this be the ultimate test? I know alot of other anti
curve settings could be done. But if in this case all tests have
passed and then after the alternate period the system collapses what
else could be done? Wouldn't this mean the whole concept of developing
systems and trading in the future would not work?
Does anyone have some suggestions or ideas about this topic?

Regards, dennis
Schutten
 
Posts: 42
Joined: Thu Oct 25, 2007 6:17 am

Postby Overload » Fri Oct 26, 2007 10:34 am

First I would say that testing 3 alternate periods is certainly helpful, but that approach has some weaknesses. In particular, with just 3 additional tests, there could still have been some degree of luck involved in seeing positive returns in those periods. As an example, flipping a coin 3 times and having it come up heads each time doesn't guarantee that it will come up heads the next time. But even further, one would need to consider how different the data from the alternate periods is to the original. If the data is all from similar "markets", then the system hasn't really been stressed by the alternate data. Instead, you're given more confidence that your system will work in that specific market condition, but you haven't learned how well your system will perform in alternate market conditions.

As for a system collapsing after it has passed all other tests, I suspect there is always some test that could have been performed that would have indicated a potential failure in that scenario. But ultimately it isn't reasonable to test against absolutely every scenario. Could the S&P suffer a 20% single-day drop? Sure, but is it reasonable to include such testing in your back test and to require your systems to be able to handle this? Most traders would call this overkill.

In the end, technical traders need to make choices regarding just how robust they wish their systems to be. We need to rely on certain assumptions, for example that the market will continue operating within the market conditions we evaluated in our back tests. If that does not happen, it was not a breakdown of our trading system, but rather a breakdown of our choice to not create a more robust trading system to begin with.

I'm often asked whether technical analysis (and therefore StrataSearch) actually "works". The answer is most definitely yes. However, it is the individual's use of technical analysis that often does not.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby taowave » Fri Oct 26, 2007 12:58 pm

Overload wrote:I'm often asked whether technical analysis (and therefore StrataSearch) actually "works". The answer is most definitely yes. However, it is the individual's use of technical analysis that often does not.Pete


Hi Pete,
I have been running backtests on the NDX 100 dating back from 1998 to the present time assuming I am encountering every type of market imaginable.I am ignoring but aware of surivirship bias.

With that said,I do find very successful systems,but the bulk of the returns are due to capturing the huge volatility early on.The latter years are T-Bill rates....

How do you view systems like that? Is there a way in OC to somehow have current returns meeting a pre defined level?? Do you typically like to backtest over periods of boom and bust??

Thanks in advance

Allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Fri Oct 26, 2007 1:52 pm

Is there a way in OC to somehow have current returns meeting a pre defined level??

In short, make use of the Alternate Data Analysis in OneClick.

Using your example, you might want to set up an Alternate Data Analysis to make sure that you keep only systems that have a worthwhile return from the years of maybe 2005 to present. That way you can be assured that your original 1998 to present works successfully AND the latter years are successful as well.

Another way to do it is to set your basic search to be from 2005 to present, but then use an Alternate Data test from 1998 through 2004.

Which you decide to do really depends on which period you want the most data for. Alternate Data Analyses are limited to a small handful of performance numbers while the full evaluation gives you everything. My personal preference is to look at everything on the most recent period, and only look at return and drawdowns for the "boom and bust" years. But that's just me.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Re: Alternate Data

Postby Dacamic » Sat Oct 27, 2007 11:07 am

Schutten wrote:... If statasearch develops a system which succesfully passes alternate period 1, 2 and 3 with minimum setting 75%. If this is the
case would not this be the ultimate test? ...

I trade a system that was developed without out-of-sample testing and continue to exclude such tests from my current development efforts. StrataSearch's brute force search is not biased by data it uses, so my belief is searches are more efficiently performed when data is kept whole rather than divided into "in-sample" and "out-of-sample" groups. From that perspective, my direct answer to your question is, "No, passing alternate period tests is not the ultimate test."

If I were to pick a metric as the ultimate test, it might be "performance consistency", and it's perfect image would be a straight line equity curve running from bottom left to top right of my monitor. Even that preferred outcome, though, must be viewed holistically by considering the entire development process. If proper consideration isn't given to factors such as sector and evaluation period selection, we have increased our odds of creating false positive test results regardless of which performance metrics are used.
Steve
Dacamic
 
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm

Postby Grek » Sat Oct 27, 2007 5:17 pm

If I were to pick a metric as the ultimate test, it might be "performance consistency", and it's perfect image would be a straight line equity curve running from bottom left to top right of my monitor.

Might it be possible to share a scoring setup that was designed with the notion of measuring this desired outcome in mind?
Grek
 
Posts: 107
Joined: Mon Sep 10, 2007 9:00 pm

Postby Overload » Sun Oct 28, 2007 8:59 am

Might it be possible to share a scoring setup that was designed with the notion of measuring this desired outcome in mind?

Yes, there's a couple things you can do. The first would be to require low yearly and monthly standard deviations. But another option that would allow even more precision would be to reference the best and worst months and require they be within a certain range. You could probably even set something up so that they need to stay within a certain variance from the Average Annual Return. That way it wouldn't be a hardcoded value, but would instead adjust itself based on the overall return.

As a supplement to the above, I would probably add a reference to "Total Active Months in Period" to ensure it makes use of a significant percentage of "Total Number of Months in Period". Consistency over time not only requires that it be successful over time, but active as well.

Pete
Overload
 
Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Postby Dacamic » Mon Oct 29, 2007 4:42 pm

taowave wrote:Do you typically like to backtest over periods of boom and bust??

I do include various market conditions in back tests. My goal isn't always, though, to determine whether a strategy or system will perform well in different conditions. For example, if a strategy runs well with the bull, i will back test it under bear conditions to see if it goes quiet when the broad market droops. So, its continued good performance isn't important, rather avoiding negative performance in unfavorable conditions is the priority.

During the past few years, the bulk of my development time was focused on finding systems suited for a cyclical bull market. It seemed reasonable to proceed in that manner since I expected bullish conditions to prevail for at least a couple years beyond when my work started. The sand may be running out of that hour glass, so my attention has shifted to finding more robust strategies by running searches from 1995 to present.
Steve
Dacamic
 
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm


Return to Curve-Fitting and Other Pitfalls

cron