Well I didn’t get any further on my plan from my last post then implementing the plug-in to PSS to dump daily stock data to CSV flat and a console application to crunch the data. Rather, then start analyzing intra-day data right away, I decided to see if I could find any metric that would have a decent correlation between past performance and short-term future performance (i.e. week, month, quarter, etc.) for a particular set of parameter values for a strategy. I’ve been getting some results from monthly data that I can’t discount yet. For quarterly analysis I think a minimum of 5 years of data is necessary get a meaningful result and weekly analysis is going significantly slower then monthly. Either way getting results is going to take a lot more CPU time, and its already running slower then I’d like it, so I need to stop and do some perf work to try to get it going faster.
The process is using less then 2MB of memory while pegging the CPU so it seems likely some caching will be helpful. The most straightforward thing to do is write my own version of the technical indicator I’m currently analyzing so that I can cache part of the work it does. For stocks with 15 years of data, caching the indicator results directly would consume 220MB of RAM; however, most of the processing work can be cached with just 3.5MB of RAM leaving just a couple of operations to get from the cached values to the full result when needed.