Diagram says it all really.
At this point I’m narrowing down dependencies and data flow.
This also has the benefit of allowing me to work in state management so that I no longer have to do chained execution and just “hope everything works”. The pieces will be able to wait on things it can’t run at the same time as, so as to avoid contention and race conditions, and will wait on status updates between the pieces.
My notes for posterity:
TDM
- needs some changes
- merge derivative data regeneration into primary refresh mode for statemanager reporting coherence
- intent is downstream services to rely on state checking and deriv symbol list instead of cache + blacklist
SIG
- add TDM state checking
- refuse or wait if it’s still running or didn’t complete successfully
= this will prevent an issue where derivative data disappears during regeneration while SIG is running
= pattern will need re-used with others potentially.
- add resume configuration execution to its daily active script before doing daily forecast
- add prune to –resume/–configure
- this will ensure all symbols forecasted after have available configurations
- add state checking for exp and sap
- contention prevention dont care about status
EXP
- add TDM state checking
- must complete successfully for LAST_TRADING DAY
- fix EXP to grab symbol list from TDM derivative data set
- re-evaluate the reporting on this, it doesn’t seem coherent/convenient with dates yet for walkbacks
- add sig state checking
- — refuse to run if sig is running, dont care about completion status, just to prevent contention
- add sap state checking
- — refuse to run if sap is running, dont care about completion status
SAP
- add TDM state checking
- must complete successfully for LAST_TRADING_DAY
- fix SAP to grab symbol list from TDM derivative data
- reporting needs reworked to be more consumable for daily processing
- data needs better compartmentalized and you need to evaluate where and how it’ll be used
- add sig state checking
- — refuse to run if sig is running, dont care about completion status, just to prevent contention
- add exp state checking, refuse to run if exp is running, dont care about completion status
MAG
- Configurations from SIG, while remarkably similar, won’t be re-useable for MAG and even with recent optimization required 20 hours to generate, because those configurations are optimized for directional accuracy not magnitude regression. This means MAG will need to generate its own configurations. It is, however, only ever operating on a subset of all market symbols — those SIG thought were going to have a positive sign (stocks where it thinks the price will go up). So, to configure the model will still take potentially dozens of hours. We’re at around 20-30 hours for SIG configuration to be safe with only about 50 real usable hours in between Friday closes and Monday opens to preload the configurations for the week. This needs to be solved.
- What I might do is introduce a passive configuration mode where MAG checks if anything else is running and just perpetually generates and updates configurations throughout the week since I’m not paying for compute. This would require checking age and availability of existing configuration. This risks screwing up rolling window testing at the benefit of pivoting more quickly to regime changes.
STS
- Not in diagram, but with the recent expansion of state checking it will be useful to have a status dashboard showing what’s running and what’s not. Maybe even a little API to query and a pretty little frontend view. Not critical but could be used as a broker or reporting service for rules-based orchestration later.
General
- these will pave the way for being able to build MAG, which will need to do SIG state checking and will pull its active symbols from SIG and not TDM
- all state checking needs to loop until ready