Validation Methodology

The objective is not to maximize one historical metric. The objective is to verify that the engine is useful under multiple stress regimes with a transparent false-alarm cost.

• Detect should-warn crises inside a fixed pre-event window
• Quantify lead time, not only hit/miss outcomes
• Measure false alarms as both day-level burden and clustered episodes
• Stress-test robustness with sensitivity, walk-forward, and ablation passes

The public contract follows the current accepted Macro Engine V4 specification. The visible MSI chart is a confirmed current-state stress display, while EWI and MTW remain separate horizon signals for fast warning and medium-term watch context.

Release governance includes validation checks and rollback controls. Those controls protect the public display contract without exposing internal gate names, trigger values, or decision parameters.

The engine separates current confirmed stress, fast warning, and medium-term watch context so readers do not treat every horizon as one blended chart.

Two policy tracks are reported to keep governance explicit:

• Operational validation policy: the policy used to evaluate responsive behavior in historical comparisons
• Stricter validation policy: a tighter regime-confirmation policy retained for continuity studies

Additional correlation and regime gates inform the research policy and telemetry paths. These include cross-asset coupling checks, macro decoupling persistence rules, and structural break confirmation methods. Exact gate logic and thresholds are proprietary.

MSI bands, alert severity, and episode labels answer different questions. Keeping them separate prevents a chart display label from being read as an alert trigger or a regime class.

Public chart rows can be muted, caveated, or absent when source coverage does not support a clean confirmed-stress reading. These states are data-quality and market-hours context, not separate stress tiers.

Each should-warn event is scored on a fixed causal protocol. Detection windows and false-alarm exclusion zones are standardized across all strategies for comparability.

Output metrics include detection rate, average lead time, and false-positive burden measured as both day-level exposure and clustered episodes per year. Exact window sizes and scoring parameters are available to partners under NDA.

False positives are measured in two dimensions because day-count alone can misrepresent operational burden.

• FP days per year captures total alert-time exposure outside crash neighborhoods
• FP episodes per year clusters consecutive FP days into discrete alert episodes
• Average episode length separates frequent short noise from rare prolonged stress regimes

This avoids a common reporting failure where one model appears cleaner only because its alerts are merged into long contiguous runs.

The benchmark suite uses 6 naive rules and 2 engine tracks under identical event windows and FP accounting.

Benchmark modes combine signals for historical comparison; the public MSI chart shows confirmed V4 stress only.

Reporting both policies makes the tradeoff explicit: operational validation emphasizes responsiveness, while the stricter validation policy emphasizes regime confirmation.

Robustness is tested through rolling normalization, walk-forward segmentation, threshold perturbation, and structural-break checks.

• Rolling quantiles and rolling z-scores are causal and use only historical observations
• Walk-forward periods are evaluated independently to detect regime-specific overfit
• Sensitivity tests perturb key thresholds to measure stability of recall and alert burden
• Structural-break testing applies repeated split tests with multiple-testing control

Ablation isolates contribution by removing one component family at a time and re-running the full evaluation stack.

Output is interpreted qualitatively as incremental value within a governance-capped family blend: if removal materially reduces detection or degrades lead-time profile, the group is load-bearing. The public page names the families but does not publish caps, formulas, or weighting schedules.

• Canonical hashing for reproducibility and artifact integrity checks
• Instrument identity gating (type, quote, venue, currency)
• Trading-day-aware freshness controls and stale-feed monitoring
• Duplicate and lag-shift detection across input streams
• Automated reconciliation gates to detect metric drift across artifacts

Historical studies answer counterfactual fit. Forward validation answers live behavior through Phase-5 validation rows anchored to Forward snapshots. Daily compatibility telemetry is retained, but operator reports read the Phase-5 row model.

• Backtest evidence is historical; live evidence is growing but still time-limited
• Some macro series have publication lags and occasional revisions
• Calibration concentration risk exists when tuning focuses on specific recent crises
• Idiosyncratic exogenous shocks can compress lead time in any model class
• No methodology can eliminate model risk; this framework makes model risk measurable

IP boundary: methodology categories, governance principles, and measured validation outcomes are disclosed; exact formulas, thresholds, weighting schedules, caps, and internal decision parameters remain proprietary.

Not advice: this system is a risk analytics tool, not financial, investment, trading, legal, tax, or compliance advice.