It’s 05:31 ET.
I’ve been at it for weeks.
We’re approaching magnitude accuracy around 70% on highly structured data. That’s pretty good, but no telling how consistent it will be until this piece I’m currently working on is done cooking.
This component is just batshit hard to build and requires insane computing power to test.
Right now I have it going on 24 processors, Ryzen 9 just maxed all the way the f*ck out for what looks like will be 11 hours. Doing per-symbol feature batteries to see which ones actually generate value and which do not so that I can prune them down to decrease configure time after deployment, because it took writing so many features just to get results better than random that it now takes 8 minutes per symbol to run a feature selection battery. That’s 11 days, so, if I had to reconfigure in the middle of a cycle, I wouldn’t be able to currently. That has to be solved.
Luckily I’m seeing not alot of features show up, so, I’m going to generate a distribution report and remove any that never get used or rarely get used. That may create openings to create better features, or, depending on the results, just remove unusable features.
MAG is almost ready for deployment after this. After that first couple weeks of finding all the things that I’ll find wrong with it and cleaning up it’ll have some forecast reports to start doing reporting on. It passes through SIG’s forecasts as well so the WIN component will only have one report it needs to read.
Updated topology/data flow:
Luckily, MAG is the hardest part of the whole puzzle. By the time it’s built, all the orchestration pieces already exist. It’s just a matter of consuming the data.
However, EXP and SAP need a revisit. Their data output is not as cut and dry as I’d like it to be. They also are not part of MVP so those can always be updated after it’s up and going. WIN is supposed to be easy since by the time the pipeline gets to MAG all the data is there and symbols just need ranked and picked.
BUY is just an API client. Nothing difficult there.
I’m almost in disbelief that it’s as close as it is. I’m about ready for the longest grind I’ve ever been on to pay off a little. If it doesn’t, it’s time for a little vacation.