It’s now late Saturday night, technically Sunday morning, and I’ve been reviewing the market conditions over the last week, and the actual performance of Tetsuo.
First, it’s doing well even though it hasn’t moved much. The DOW was down 600 points at one point and it made some money, and then when it should have bottomed out it held its value. That’s a good sign about its forecast ability, but it does point to a potential inventory health issue.
‘Lo and behold, TDM was set to only pull from NYSE and completely skipping over NASDAQ symbols. This was an early concession I made over a year ago for testing that I never came back and resolved. Not a huge issue, but, it means it was getting “okay” results with only maybe a 3rd of its symbols being usable and only maybe 2/3 of the symbols it should be looking at.
To top that off, symbols that SIG and MAG did feature set training on, had about 500 symbols that had low accuracy due to junk data (for junk stocks) and while those would not have made it into the forecast reports due to being filtered by accuracy, they do consume time and resources being forecasted on every day.
There is also a symbol blacklist that I keep adding symbols to that either don’t have enough history, or don’t trade enough volume to be perform gradients against, and between all of those, that’s capacity freed up.
I’m hesitant to expand the available inventory, due to how close it is currently to not having the hardware needed without great expense. I think it can realistically handle about 3,000 symbols, last run was on about 1800, and the addition of NYSE makes it about 5400.
So, I’d need filters. For symbols whose price dropped below a dollar recently. For symbols with shit volume recently. For symbols with missing days of data. Then I’m screwing around constantly with TDM if I do that, which handles primarily historical data.
So, to solve this, I’m decoupling the function of inventory management from TDM and I’m building a new component that manages the inventory for TDM to pull from. For now I’ll call it UIP (Universe Inventory Provider). After that goes up I’ll need to make some changes to TDM to use it instead of connecting directly to the exchange-linked providers.
It all needed another pass for quality rewrites anyway since it’s a new system, so, it’ll be fun to keep it up and running as is while incrementally introducing the changes I want to see in it. I’m already mostly through testing a draft of UIP, so, this may go up as early as tonight.
Overall, still quite pleased with how it’s going after the big MAG hurdle even despite the rough landing. That may change Monday, I reviewed its winners list for Monday and we’re expecting some volatility, which may be really good but could also be really bad. I’ve also noticed that when it catches big swings up, it’ll almost always catch a big swing about that much in the negative direction and they cancel each other out. Maybe this filter will help with that since it’s not doing enough data quality testing yet, but, it’s most assuredly not going to be in place by Monday, and I’m expecting a busy week at the dayjob.
As I type this it just finished. 4,281 symbols. The filtering took like 40 minutes since it’s doing some data inspection. That means TDM will take twice as long as it has been, so will MAG and SIG, if I bring it in. But I kind of need to if I want to make more money, more reliably.
So, what is that. Well, TDM Derivatives take 27 minutes to generate. Let’s call it a half hour. Primary even in parallel mode takes 2h 30m (and 44 seconds) with 2423 symbols, averaging about 16 symbols/min. That’s if it doesn’t need to configure which takes 4 times as long. That includes a ton of failures, though, from like data being incomplete, or a symbol having not been on the market long enough to have the data to train on — which this change would remove alltogether. MAG takes about 40 minutes. So, let’s say I cook feature configuration at home where I’ve got more firepower, and preload those, give it some buffer, that’s 6.5-7 hours for TDM, call it 7 to be sure, 80 minutes for MAG, let’s call it 90. 2 hours for SIG turns into 4 hours. So, it has to start at like 1715 and has until 0900. That’s 15.75 of availiability. 12 and a half hour runtime with a big wide swing depending on market conditions. It’s doable, but it’s pushin’ it too close. I’ll need to look closer at the filters before bringing it in.
Time to stew on it and do other stuff for a while unless Monday turns sour on it.