Prevent duplicate checks during oneclick searches

Let us know your likes and dislikes, or what you'd like to see next. We listen!

Prevent duplicate checks during oneclick searches

Postby Jaded » Sat Jun 25, 2011 4:44 pm

I'll try to explain my problem:

I am having an issue when running a Oneclick search on a large sector (2000+).. It's probably not as noticeable on a small sector, but I think it should be addressed.

Scenario: Oneclick is searching for strategies using only 1 additional rule (in addition to a primary entry/exit rule).. That rule does not have many parameters (maybe 1 or 2).. My problem is that it keeps trying the same exact parameters over and over again for 250-500 runs (whatever it is configured to) before moving on. This slows down processing a lot, especially with alternate sector data that it keeps checking whenever it thinks it has found something worthy of checking alternate data of.

Example Entry String:
// Any primary rule entry rule here
(close > 1) and
(mov(volume, 20, simple) * close >= 200000) and
(qst(16) < 0)

Example Exit String:
// Any primary rule exit rule here

--- The oneclick keeps searching over and over again for values in qst that are between 10 and 20 (by 2.. so 10,12,14,16,18,20).. It does this for 250-500 runs as I said, and if the system is close to passing all parameters, this ends up in a very long process to make it past the rule.

My suggestion, which I think it simple enough.. combined the entry/exit strings into a single string.. MD5 that, and push it onto an array. Keep an array of the last 500 (or whatever #) md5'ed strings. If the MD5'ed current Entry/Exit string it is going to attempt matches one of those existing md5 hashes, skip testing that entry/exit string... this will drastically speed up processing and eliminate a lot of duplicate checking for these rules that are really good, but just don't have enough parameters to warrant testing for hundreds of runs, if it is just going to keep checking the same thing over and over again...

This will also prevent me from getting 5 strategies in my multi-system that are EXACTLY identical entry/exit/rank strings (didn't know it would do that.. but it does).

Let me know what you think of this suggestion.. I'll also take this opportunity to ask for an x64 version, just for the memory size limitation for those of us that run huge sectors and have 8+ core processors.. there will soon be 24+ core computer systems out there that will end up depriving Stratasearch of RAM very quickly when you have 24+ threads each using the amount of RAM necessary to perform its calculations.. just a thought :)

Posts: 151
Joined: Mon May 14, 2007 6:49 pm

Re: Prevent duplicate checks during oneclick searches

Postby Overload » Sun Jun 26, 2011 1:15 pm

I definitely understand the issue you're talking about, although I've run into it more when running Dynamic Strategies with a limited number of rules and I want a more efficient use of the search time.

Currently, the protection against running the same strategy twice has been whether it exists in the Combination Results Listing or not, although this does not apply to OneClick Searches. This has limitations since the space it takes to store a combination result is actually quite large, and it is a relatively slow process to index and retrieve them when displaying the Combination Results. That's why there's a current limit of 50,000 results in each Results Database.

I've recently thought about your MD5 idea as well, and I agree that MD5 or some other checksum would probably be the best approach. We would actually need to include all of the selection fields in there, including Sector, Rank and Order Types/Prices. But to be able to combine that into a 4 byte value for each system would make it efficient enough to store large numbers of them. To do it correctly, I would prefer to store those values on the disk so that a user can always shut StrataSearch down and restart exactly where they left off. In any case, thanks for the suggestion. I will put this on the list (or, rather, bump it up a couple notches).

As for the 64-bit, I agree, and that is also on the list. It's a somewhat larger enhancement primarily because StrataSearch uses a variety of 3rd party modules, which will also need to be 64-bit for full compatibility (and benefits). And the problem is that 64-bit versions of those modules aren't always available, so entirely new versions of those modules need to be found, purchased, implemented, tested, etc. It is unfortunately a lengthy process.

On a side note, the rumors I'm hearing are that Windows 8 will be 128-bit. At some point it is likely that Windows will stop supporting 32-bit applications, so a StrataSearch upgrade to 64-bit (and 128-bit) will definitely happen at some point.

Posts: 2248
Joined: Wed Nov 30, 2005 12:14 pm

Return to Comments and Suggestions