One Clicks and random thoughts

StrataSearch will find what you tell it to, but what should that be? Will the same criteria work across different Sectors? Different time periods? Here we discuss the ins and outs of OneClick Searches.

One Clicks and random thoughts

Postby taowave » Sun Sep 09, 2007 6:21 pm

Hi all,
I have predominantly used AS in SS,but as I have expanded my testing to include various market conditions,i.e 1999-2007,it appears that "non adaptive" probably doesnt quite cut it.I noticed in most of my backtest results,the systems performed very well early on,and tapered off significantly in the last 2 years..Am I correct that in theory,system selection will be biased to systems that perform well in the early time periods?? i am thinking along the lines that an initial 50% loss requires a 100% gain to get back to flat....

Pete/Steve,may I ask if you lean towards OC in your trading and would you be kind enough to breifly explain why?? For the record,I am becoming an OC Dynamic strategy convert


thanks,

allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Mon Sep 10, 2007 10:37 am

Whether one uses OneClick, AutoSearch, or codes their own individual Strategies is entirely personal preference. I can't say that one is better than another because it really depends on what one is looking for.

My personal preference is using the OneClick search to look for trading systems, and that is for a couple reasons. The first reason is time. I just don't have enough of it to sift through the hundreds/thousands of results one needs to sift through when running an AutoSearch. So I would rather just let my computer do the work while I tend to other things. Granted, not everything the OneClick finds is worth keeping. But I can just discard the ones that aren't worth keeping and restart the search. This takes very little of my time since I only have to examine a small handful of results every few days rather than hundreds every few days.

The second reason I prefer the OneClick is that I want to leave very little to my own discretion. Having proved to be an exceptionally poor trader when left to my own trading decisions, I now want everything I do based on mathematical backing. The OneClick lets me (and in fact forces me) to put everything I want in a trading system into a mathematical formula. This can be percentage profitable, drawdowns, consistency over time, alternate data evaluations, parameters shifts, etc. By placing these "filters" into my search in the first place, I can very specifically identify strategies that match what I'm looking for. Ultimately, this saves me time, but it also keeps me from having to make personal judgements regarding whether a strategy meets my needs or not. Sure, I still need to make personal decisions on what characteristics I'm looking for in a strategy. But for me this is much easier than manually comparing hundreds of performance values between hundreds of possible strategies. I admit being overwhelmed having to compare and rank hundreds of strategies. And that's why I'd much rather have the OneClick Score do that for me. I just enter my scoring algorithm and filters and I let the OneClick handle the rest for me.

As for using Dynamic Trading Strategies (DTS), I haven't had as much luck with those as I'd hoped. As this functionality was being implemented, it seemed like a very powerful idea. If you could regularly build new strategies based on the most recent data, you would always be taking the greatest advantage of the most recent market conditions. And if you could back test this approach using DTS, you could confirm and even enhance your selection methodology. But after some significant tests, I was never able to find an approach that ultimately worked better than a static system (except for one method I'll mention below). What I discovered was that, by only using the most recent data, there simply wasn't enough history in the back test to ensure the selected stategies were robust enough. Thus, I was finding that they would then fail moving forward in the DTS back test.

I should mention, however, that most of my testing with DTS was done before the Alternate Data and Parameter Shift tests became available. So there's reason to believe the problems I was having could now be compensated for. For example, if one were to not only use the most recent data to create the DTS strategies, but also comfirm their robustness by running an Alternate Data analysis on a larger evaluation period, one could theoretically get the best of both worlds. I have not explored this yet, but it seems like a logical idea.

The one DTS approach that I now use regularly is called a Walk Forward Analysis (WFA), and is described in a book by Robert Pardo called "Design, Testing and Optimization of Trading Systems". This was recommended by another StrataSearch user, and I've found the concept to be quite helpful. The basic idea is that your trading system itself is static, but the parameters themselves should be re-optimized periodially over time. Thus, I take a strategy I'm interested in, set up a DTS so that only that strategy will be used (with variables set in each of the parameters), and then run it through a large evaluation period. As the parameters are then adjusted every 2 months based on the most recent data, and then tested on their respective walk-forward periods, I can then see how this system performed with this dynamic approach over a period of several years. If the system performed consistently well in this back test, the theory is that the system can adapt. And this gives confirmation that the system is both robust and well-fitted. In general, I have seen that about 1 in 5 of my strategies meet this WFA challenge, but this is largely dependent on my initial strategy filters. And while I don't believe a WFA test is a necessity for a solid strategy, it's a test I currently like to perform. As I said, a mathematical confirmation is very important to me. And this one, I think, is a pretty good one.

That's my 2 cents. I think Steve, however, is still an AutoSearch fan. Hopefully he'll throw his 2 cents in as well.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Postby taowave » Mon Sep 10, 2007 12:26 pm

Hi Pete,
That was excellent and very generous of you.Thanks for sharing..

Pete,I have another question regarding the Annual Returns in Combination results vs Detailed analysis.For simplicity,I will apply the questions to a Strategy and fixed trade equity

Am I correct that Rank Selection applies to both Combination Result and Detailed Analysis??

The returns for Combination Results are averaged annualised returns of a system and NOT reflective of the actual sequence of trades nor the equity in an account,i.e every trade is taken...

I am only asking this as it dawned on me that systems/system selection will be largely influenced by the sequence of trades,especially when a large drawdown occurs early on...

I take it this is why one could have vastly different returns for Combination results vs detailed analysis performance(all other things equal),and why I should probably incorporate Monte Carlo returns..

Thanks again

Allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Mon Sep 10, 2007 3:18 pm

Am I correct that Rank Selection applies to both Combination Result and Detailed Analysis??

It applies only to the xPortAPR numbers in the Combination Results. The other columns in the Combination Results are not portfolio specific.

The returns for Combination Results are averaged annualised returns of a system and NOT reflective of the actual sequence of trades nor the equity in an account,i.e every trade is taken...

No, the xPortAPR numbers are reflective of the actual sequence of trades. They are not, however, based on a flow of equity but instead reflect the combined profits from all portfolio-based trades. The xPortAPR numbers are based on the Trades Report in the Detailed Analysis, which does not include margins or interest. But also, unlike the Equity reports, it will not bypass any trades due to a lack of equity available to make a purchase. The portfolio selection, however, is still used.

I am only asking this as it dawned on me that systems/system selection will be largely influenced by the sequence of trades,especially when a large drawdown occurs early on...

Yes, in the Equity Reports, that is particularly true. If you examine a portfolio size of 10, which loses 30% early on, you'll have to recoup those losses with your remaining 7 pools of cash. Naturally this will take longer, but it properly reflects what would happen in real trading without injections of additional cash to your portfolio. Again, you can also examine the Trades Report, which will continually keep your 10 pools running despite any losses early on.

I take it this is why one could have vastly different returns for Combination results vs detailed analysis performance(all other things equal),and why I should probably incorporate Monte Carlo returns..

More specifically, it's why you can have vastly different returns for a Trades Report approach versus an Equity approach. Using an Equity approach, the portfolio will shut down if there isn't enough cash to continue trading at that level. The Trade Report approach, however, will continue churning away regardless of the degree of losses. Neither approach is better or worse. They are just different views of the account activity.

It's also important to mention that both the Trades and Equity Reports will purchase the exact same portfolio-based selection of trades (assuming no trades are bypassed due to a lack of equity). In other words, there may be a whole group of valid signals out there that you may never see when looking at the system from a portfolio perspective. But it is the Monte Carlo analysis that will allow you to see those non-portfolio trades. And that's why the Monte Carlo is indeed an important perspective.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm

Postby taowave » Mon Sep 10, 2007 10:46 pm

Yes, in the Equity Reports, that is particularly true. If you examine a portfolio size of 10, which loses 30% early on, you'll have to recoup those losses with your remaining 7 pools of cash. Naturally this will take longer, but it properly reflects what would happen in real trading without injections of additional cash to your portfolio. Again, you can also examine the Trades Report, which will continually keep your 10 pools running despite any losses early on.


Interesting,and now its becoming bit clearer as to the value of Alt data series and running a system starting with a major market peak,i.e 3/2000..

Thanks Pete
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Re: One Clicks and random thoughts

Postby Dacamic » Tue Sep 11, 2007 5:13 pm

taowave wrote: ... as I have expanded my testing to include various market conditions,i.e 1999-2007,it appears that "non adaptive" probably doesnt quite cut it.

I have found the same to be true in my experiences to date. I've recently started a search to further explore this area, but it's too early to know whether those efforts will produce a different result.

I noticed in most of my backtest results,the systems performed very well early on,and tapered off significantly in the last 2 years..Am I correct that in theory,system selection will be biased to systems that perform well in the early time periods?? i am thinking along the lines that an initial 50% loss requires a 100% gain to get back to flat....

There is a long-winded answer to this topic, which I'm desperately trying to resist writing. A shorter, probably more valuable answer follows:
    1. It is not uncommon for AutoSearch to create systems whose profits are clustered within one portion of an Evaluation Period;
    2. It is not uncommon for AutoSearch to create systems with significant early drawdowns; and,
    3. When profit-based Combination Filters are set, systems with significant early drawdowns often do not pass those filters, while systems with clustered profits do; accordingly, we are more likely to see the latter than the former in our results.
Pete/Steve,may I ask if you lean towards OC in your trading and would you be kind enough to breifly explain why?? ...

I usually use AutoSearch for my searches. Heck, even my OneClick searches are configured to run very similar to AutoSearch.

AutoSearch and OneClick use identical processes for initially creating and back testing strategies; thus, neither search is inherently better than the other at creating profitable strategies ('cuz the strategies created by AutoSearch are essentially the same as those created by OneClick). The choice between the two therefore mostly depends upon the level of automation that is preferred for tasks downstream from back tests.

A OneClick search is a little more difficult to set-up and maintain (compared to AutoSearch), an understandable side effect of its more extensive use of automation. Possibly because of that added complexity, I've not been able to create a work flow using OneClick that is as comfortable to me as the one I have with AutoSearch. If I had "grown up" using OneClick instead of AutoSearch, it is certainly possible my preferences in this regard would be different.

To paraphrase a comment you made several months ago, Allan, whether to use AutoSearch or OneClick really boils down to deciding which tasks you will perform and which you want to have done by your "research assistant". Ultimately, it probably takes some experimentation with various workloads to find a balance that works best for each trader/research assistant team.
Steve
Dacamic
 
Posts: 457
Joined: Wed Nov 30, 2005 12:40 pm

Postby taowave » Tue Sep 11, 2007 8:06 pm

Hi Steve,
Appreciate the insight..The more I think about it,the more i realize I created this dilemna.

I have expanded my testing to include various market conditions,i.e 1999-2007,it appears that "non adaptive" probably doesnt quite cut it.I noticed in most of my backtest results,the systems performed very well early on,and tapered off significantly in the last 2 years


I just happened to choose a starting period that was the peak before a 50% decline in the Dow,and probably much worse in the Nasdaq.Obviously it was an extreme case which i had never encountered.

You are dead right regardng AS vs OC.I personally find the Static vs Dynamic debate a fascinating topic,and my efforts would be better spent there..

Allan
taowave
 
Posts: 584
Joined: Sat Dec 02, 2006 12:39 pm

Postby Overload » Tue Sep 11, 2007 10:22 pm

You are dead right regardng AS vs OC.I personally find the Static vs Dynamic debate a fascinating topic

Just to prevent confusion from others that might read this thread, it's important to note that the difference between OneClick and AutoSearch isn't that one creates dynamic and the other static systems. In fact, the Dynamic Trading Strategies feature of OneClick is just a small, optional feature of that mechanism. Normally, OneClick is used to create static systems just like the AutoSearch does. OneClick happens to add a high-level filtering mechanism, along with other features like scoring and optimization to the otherwise unintelligent AutoSearch. But a OneClick Search has the ability to search for either static or dynamic systems. It all depends how it's configured.

Pete
Overload
 
Posts: 2246
Joined: Wed Nov 30, 2005 12:14 pm


Return to OneClick Searches and Scoring

cron