Well, with the results in for 16 out of 22 stocks, 3 metrics are standing out for both the last 2 years and the last 5 years worth of history for those stocks. It will be sometime tomorrow until the full results are in, as I promptly added more analysis after my optimization work.

As further decreasing runtime via manual optimization is going to be quite development time consuming relative to the runtime saved, I decided to research and purchase the parts to put together a quad core machine. For $475 and the time it probably would have taken to optimize enough increase throughput another 10%, I’ll have a machine with twice the throughput and I can free up my primary machine so I can use it exclusively for doing new development work. The analysis runtimes are getting long enough that I may be able kick off a run, do the development work for the next run after that, and still have time to some work on my game before all of the results are in.

I’m also thinking about making the analysis tool into a distributed app. Currently it isn’t even multi-threaded since the proc can be fully utilized by running analysis on different stocks in parallel, but this requires a bit of manual intervention on my part to keep things fully loaded. By making it fully distributed I can also still have my primary machine automatically start do analysis work when its idle and I can also then easily added more machines if necessary. The distributed app infrastructure should also be largely shareable with my game as well and it will give me an excuse to look into Google’s recently released Protocol Buffers data interchange format, which is appealing as it supposed to be simple and its compiler produce code for both C++ and Python (and Java).

The partial caching of the calculations of the technical indicator resulted in a 35% less runtime and experimentation with the various complier and linker optimizations switches resulted in a little be more of a speed up for in runtime being an overall of 39% shorter.

The analysis code likely has some spots that will benefit from optimization; however, that is currently in flux, so it isn’t worth hand optimizing at present.  So its back to flushing out a useful metric.

We’ll I didn’t get any further on my plan from last post then implementing the plug-in to PSS to dump daily stock data to CSV flat and a console application to crunch the data. Rather, then start analyzing intra-day data right away, I decided to see if I could find any metric that would have a decent correlation between past performance and short-term future performance (i.e. week, month, quarter, etc.) for a particular set of parameter values for a strategy. I’ve been getting some results from monthly data that I can’t discount yet. For quarterly analysis I think a minimum of 5 years of data is necessary get a meaningful result and weekly analysis is going significantly slower then monthly. Either way getting result is going to take a lot more CPU time, and its already running slower then I’d like it, so I need to stop and do some perf work to try to get it going faster.

The process is using less then 2MB of memory while pegging the CPU so it seems likely some caching will be helpful. The most straightforward thing to do is write my own version of the technical indicator I’m currently analyzing so that I can cache part of the work it does. For stocks with 15 years of data, caching the indicator results directly would consume 220MB of RAM; however, most of the processing work can be cached with just 3.5MB of RAM leaving just a couple of operations to get from the cached values to the full result when needed.

At this point what I’ve developed is a prototype for a strategy curve fitting detector and thus far the technical indicators I’ve tried all just fit the curve of the price history. Basically I’m finding that while a backtest for a strategy utilizing an indicator with a given set of parameters for the indicator may show triple digit annualized returns, if the first year of results are compared to all of the time after the first year, there is very little correlation between the first year’s results and the following years results, such that if you were actually sitting back at the end of that first year, you would not be able to pick the parameter set for strategy based on the results it produced. If it couldn’t be done then, then its reasonable to assume it can’t be done now either. So no viable strategy, but at least now I’ve got a good way to reject strategies quickly (relatively speaking it still may take hours of CPU time per symbol depending on how big the parameter space is for the strategy).

Now I need to clean-up the mess of code I wrote getting to this point. First I want to create a plug-in to Personal Stock Screener that will dump historical daily and intraday data into a CSV flat file, which can then be used as input for a console app that will load the strategy from a DLL, so I can invalidate strategies in a process different then PSS, although I’ll still use PSS to update the flat files daily. I also need to get Sierra Charts as with the right subscription level it provides 6 months of historical intraday data. If that isn’t enough, then I’ll have to buy it from one of the few providers on a per symbol per month basis.

Before discounting technical indicators and moving on to Elliot waves, Fibonacci sequences, fractals, and perhaps neural nets I want to run them over intraday data as I got some very interesting results with an assumption that I couldn’t validate as correct with daily data, but will be able to validate or invalidate with intraday data.

After investigating a number of alternatives (one, two, three) to NinjaTrader, I ended up implementing my strategy as a COM plug-in to DTLink’s Personal Stock Screener, and found that I can run through the equivalent strategy optimization, as in NinjaTrader, 255 times faster.  A significant part of the speed improvement likely comes from three orders of magnitude less memory usage during the optimization process (comparing each process’s increase in memory during optimization, not total process memory usage).

TA-Lib proved quite useful for technical analysis functions and apparently Personal Stock Screener uses TA-Lib as well.

Anatoly Ivasyuk, at DTLink, was very responsive to the few questions I had, as well as quickly turning around the fix for a bug I ran into.

Since opentick isn’t currently accepting new users I’ve been looking into other free and easy ways of getting historical and real-time data to feed a strategy written in C++ against TA-Lib.

CoolTrade

CoolTrade is another trading platform. This one carries a subscription fee, but it is free to TD Ameritrade client’s. It does allow strategy creation, but only via a UI. No coding allowed. From their forum I gather that backtesting is supported, but only fro TD Ameritrade data feeds and its still in Beta. Didn’t get the impression that strategy optimization was supported either, so this isn’t what I’m looking for either. It also didn’t seem provide charting, but the UI was quite non-standard and I simply have not be able to find the way to get a chart up.

Personal Stock Streamer

There is also Personal Stock Streamer and Personal Stock Monitor, which, although they have separate web sites, are both DTLink Software products and there is a least a fourth web site run by DTLink. Personal Stock Streamer is a free download. The Standard and Gold editions of Personal Stock Monitor costs $30 and $50 respectively (one-time not subscription), with a 40% discount for TD Ameritrade client’s and there is a 30 day free trial. The said, it wasn’t immediately obvious what the difference was between DTLink’s Streamer and Monitor. The configuration process, overall UI, and Options->Preferences appear nearly identical. They also appear to provide an identical set of indicators. The set of indicators provided is smaller then TD Ameritrade’s StrategyDesk, QuoteTracker, and NinjaTrader, but lager then open source iTrade and XTrader. Like QuoteTracker, neither one shows the value of indicators for the current bar in the data box where price and volume are shown. The most obvious difference I could find is that Personal Stock Streamer is extensible via OLE/ActiveScript, but, as far as I can tell, Personal Stock Monitor is not extensible in anyway, which is odd considering the former is free and the later is fee based. At this point in time I can see no reason to go with the fee based Personal Stock Monitor over the free Personal Stock Streamer.

Since the scripting object model is provided via OLE, it is technically callable from C++ even though no header files are provided and it isn’t officially supported or documented (there is typelib in the SDK though and I know a tool exists to create a header from a TLB). That said, it looks like extensions must use ActiveScript as extensions have to be installed and they need to be formated as an XML file with the script in the CDATA section, but I’d imagine the extension script could just instantiate a COM object and delegate everything to it. I also found a two threads on their developer forum that mentioned an SDK/API that would allow developing a COM plug-in, but the threads were from late 2005 and then it was necessary to e-mail them to get it. I’ve done that, so we’ll see if that plug-in SDK is still available.

Sierra Chart

Sierra Chart is extensible in C++ and has numerous different subscription levels and discounts. It looks like it will run me $17.50 a month or $175.35 a year for the cheapest level that supports TD Ameritrade. They appear to have a 7 day trial, so before trying them out, I’d want to actually have a strategy written against TA-Lib, so I could get integrated into Sierra Chart right away and determine if its worth it for the long term, so I can sign-up for a longer time period and save some coin if it proves worthy.

QuoteTracker API

I mentioned QuoteTracker previously, and while I didn’t look at it very long then, today I discovered that it apparently has an API that may allow me to get historical and real-time data from TD Ameritrade and automate trading.

TD Ameritrade API

TD Ameritrade actually has their own API; however, they don’t give it out to just everyone, and I don’t trade any where near close to the necessary frequency to qualify.

NinjaTrader Proxy Strategy

Another way to get the historical data would be to create a strategy in NinjaTrader that simply dumps the data into a flat file or SQLite DB, which could then be processed any other process. Real-time data could be had and trading could be automated by creating a strategy that serves as a proxy for other processes to NinjaTrader.

I just tried to sign-up for opentick and was greeted with “opentick is upgrading its network infrastructure and temporarily cannot accept new users.” Checking their forum I found that they haven’t been accepting new users for 2 months now and their is no ETA for when they will be accepting new users. Before that I also found that using the opentick Python bindings would require figuring out how to build both the bindings and the SIP project on the Win32 platform, so I had figured on just going with C++ to get something going right away. But, since I can’t sign-up that’s a no go.

iTrade is an open source (GPL) trading platform written in Python. At present it provides a few indicators on the charts, but these are not configurable (i.e. you can’t specify the period). The only data feed provided (for US markets) is from Yahoo. It also does not presently support the development of strategies, so no backtestings or optimization either, so this is not what I’m looking for.

XTrader is also an open source (GPL2) trading platform, but it is written in C++ rather then Python. This also provides a few configurable indicators and the only data feed provided is from Yahoo. It also does not presently support the development of strategies either.

I found both of these on the website for TA-Lib.org, which is the home for a C++ technical analysis library. That site has an extensive list of open source projects related to technical analysis, and I did not try them all, just the two above.

At this point I think my best bet will be to write a strategy in C++ that makes use of TA-Lib for common indicators and use opentick as my data feed. Now I just need to determine if the entire app should written in C++ or just the strategies and indicators, with the rest being written in Python. The first thing to determine is how straightforward the opentick official C++ API is versus the unofficial Python API.

After installing NinjaTrader on my secondary machine, which isn’t as beefy as my primary machine, the first optimization run hung, and the second two attempts both silently crashed (i.e. NinjaTrader just disappeared). I had the task manager up for all three runs and it didn’t even break 300MB of memory usage, so excessive memory use doesn’t appear to be the problem, as it was on my primary machine when the trace/log files got to bloated. On my primary machine I had to uninstall it and reinstall it to get the strategy drop down populating again in the backtesting optimization configuration dialog (repair didn’t work).

So that means I can only run it on my primary machine. If it builds up giant log/trace files again and starts using up 800MB of RAM just to startup again, then it’ll will again be necessary to purge these files, but its unclear if this can be done without doing a full uninstall and reinstall.

Anyway, I’ve started looking at other options even while I continue to run it on my primary machine, and I’ll switch over to something else as soon as I find it or write it.

Yesterday I started running into problems with NinjaTrader not completing optimization runs. After several tries, including after closing down re-opening the app, I started monitoring the app more closely and the first thing I noticed was that it was consuming large amount of memory and seemed to quit sometime after going over 1GB of usage. Turns out it was consuming over 800MB just starting up. Tried lightening up what was in the workspace, but that didn’t help. Next had a look at what was in its data directory and noticed it has some pretty large files under dirs called log and trace, so I figured it was probably fully loading these into memory and deleted them. That got the start-up memory back down to about 75MB, which more reasonable for a .Net app. Everything looked good until I tried to run the optimizer again. It’s got a drop down box used to select which strategy to run, but its empty. Other parts of NinjaTrader show the strategies just fine, but apparently something related to that drop down has a dependency on one of the files, so It looks like I’ll need to get something restored to make it happy again (didn’t bother backing up the files).

Oh well, I was going to install it on another machine so I wouldn’t just let it run optimizations there over the course of days without it interfering with my use of my main machine.