Validation Methodology

The objective is not to maximize one historical metric. The objective is to verify that the engine is useful under multiple stress regimes with a transparent false-alarm cost.

• Detect should-warn crises inside a fixed pre-event window.
• Quantify lead time, not only hit/miss outcomes.
• Measure false alarms as both day-level burden and clustered episodes.
• Stress-test robustness with sensitivity, walk-forward, and ablation passes.

The production stack is MSI + EWI + MTW. MSI captures current stress state. EWI captures trajectory risk. MTW stabilizes output behavior across short-term noise.

Two policy tracks are reported to keep governance explicit:

• Live production mode: the operational pipeline used for real-time alert generation.
• Research-calibrated mode: a stricter regime-confirmation policy retained for continuity studies.

Additional correlation and regime gates inform the research policy and telemetry paths. These include cross-asset coupling checks, macro decoupling persistence rules, and structural break confirmation methods. Exact gate logic and thresholds are proprietary.

Each should-warn event is scored on a fixed causal protocol. Detection windows and false-alarm exclusion zones are standardized across all strategies for comparability.

Output metrics include detection rate, average lead time, and false-positive burden measured as both day-level exposure and clustered episodes per year. Exact window sizes and scoring parameters are available to partners under NDA.

False positives are measured in two dimensions because day-count alone can misrepresent operational burden.

• FP days per year captures total alert-time exposure outside crash neighborhoods.
• FP episodes per year clusters consecutive FP days into discrete alert episodes.
• Average episode length separates frequent short noise from rare prolonged stress regimes.

This avoids a common reporting failure where one model appears cleaner only because its alerts are merged into long contiguous runs.

The benchmark suite uses 6 naive rules and 2 engine tracks under identical event windows and FP accounting.

Reporting both modes makes the tradeoff explicit: live policy emphasizes operational responsiveness, while research-calibrated policy emphasizes stricter regime confirmation.

Robustness is tested through rolling normalization, walk-forward segmentation, threshold perturbation, and structural-break checks.

• Rolling quantiles and rolling z-scores are causal and use only historical observations.
• Walk-forward periods are evaluated independently to detect regime-specific overfit.
• Sensitivity tests perturb key thresholds to measure stability of recall and alert burden.
• Structural-break testing applies repeated split tests with multiple-testing control.

Ablation isolates contribution by removing one component family at a time and re-running the full evaluation stack.

Output is interpreted as incremental value: if removal materially reduces detection or degrades lead-time profile, the group is load-bearing.

• Canonical hashing for reproducibility and artifact integrity checks.
• Instrument identity gating (type, quote, venue, currency).
• Trading-day-aware freshness controls and stale-feed monitoring.
• Duplicate and lag-shift detection across input streams.
• Automated reconciliation gates to detect metric drift across artifacts.

Historical studies answer counterfactual fit. Forward validation answers live behavior. Predictions are logged daily and scored only when sufficient future data exists to evaluate the outcome.

• Backtest evidence is historical; live evidence is growing but still time-limited.
• Some macro series have publication lags and occasional revisions.
• Calibration concentration risk exists when tuning focuses on specific recent crises.
• Idiosyncratic exogenous shocks can compress lead time in any model class.
• No methodology can eliminate model risk; this framework makes model risk measurable.

IP boundary: formulas, governance, and measured outcomes are disclosed; exact thresholds, weighting schedules, and internal decision parameters remain proprietary.

Not financial advice: this system is a risk analytics tool, not an investment recommendation engine.