Mo Rahmani / Research

FX Directional Forecast — Backtest Methodology

A rolling out-of-sample evaluation framework for binary directional models across multiple FX pairs and timeframes.

← Back to portfolio Selected research Discuss

1. Overview

This project produces rolling out-of-sample (OOS) directional forecasts for FX pairs across daily, weekly, hourly, and 4-hour bars. The core question is: given everything knowable at bar close t, can a non-linear classifier predict whether the next bar closes higher or lower than the current one?

The pipeline is intentionally conservative: all model selection decisions are made on data strictly before the OOS window, no future information is used in feature construction, and performance is measured only on held-out steps.

Raw data FX OHLC FRED macro Feature panel Technicals PCA (k=5) Rolling OOS Tune on inner split 25-bar OOS batch Sticky champion Deploy Refit on latest Predict t+lead Report
End-to-end pipeline from raw market data to daily forecast report.

2. Data inputs

Market data

Daily FX OHLC bars are sourced from FXCM public candledata (bid prices), with Yahoo Finance fills appended for any missing tail through the current calendar date. This produces a continuous series from approximately 2000 onward for major pairs.

An optional synthetic "today" bar is injected after New York 5 PM close (FXNL_INJECT_LIVE_QUOTE), allowing same-day forecast generation before official bar close.

Macro fundamentals

A macro panel is drawn from the FRED API: rates, yields, spreads, and broad indices from 2000-01-01. The panel is built at daily frequency and forward-filled to align with FX bar dates. Key rationale: macro regime features provide cross-asset context that pure price-based models miss, particularly around rate differentials and yield curve dynamics.

Universe

The default universe covers 10 major FX pairs (G10). Extended sweeps additionally include non-FX instruments (equity indices, commodities) as context features or standalone forecast targets.

Data hygiene note: all feature columns are computed from data available strictly at bar close t. Forward returns used as labels are derived from close-to-close moves at t+lead. No lookahead is introduced in the feature construction step.

3. Feature panel construction

For each primary pair the modeling frame is assembled as follows:

  1. Merged panel — load primary pair OHLC and FRED macro columns into a joint date-indexed frame.
  2. Cross-pair context — for each context pair in the universe, join only its prefixed columns (e.g. EURUSD_rsi_14). This brings in correlated pairs' momentum and volatility state without duplicating raw prices.
  3. Drop raw closes — all *_close columns are removed from X. The model never sees the raw price level, only derived features. This prevents trivial level-based leakage.
  4. Rolling volatility — GARCH-style rolling volatility columns are added per pair to give the model explicit regime context.
  5. Label construction — binary target: y = sign(close[t+lead] / close[t] − 1). Ternary variant adds a neutral band (±0.5% threshold). The forward return forward_r is retained separately for strategy-level evaluation.

PCA pre-processing

Within each rolling OOS window the feature matrix is pre-processed identically to the backtest pipeline: median impute → StandardScaler → PCA (k=5 components) → re-scale. This reduces the effective feature space and mitigates multicollinearity from the high-dimensional technical/macro panel.

PCA variance analysis confirms that 5 components typically explain a meaningful share of cross-sectional variance in the merged technical + macro feature matrix, with marginal gains from additional components. The k=5 default is validated by a dedicated sweep over k ∈ {1, 2, 3, 5, 10, 15, 20}.

100% 50% 0% PC1 PC2 PC3 PC4 PC5 PC6+ Cumulative var. explained
Indicative PCA scree profile — cumulative variance explained by component count. Actual values vary by pair and timeframe; the k=5 default captures the dominant structure while avoiding overfitting to noise components.

4. Rolling OOS backtest design

The core evaluation uses a rolling batch holdout scheme, not a simple train/test split. This is the most important methodological choice in the pipeline.

t Expanding history pool Inner train (200 bars) Inner test (50) OOS batch (25) Next batch … t₀ Batch origin Re-tune Tune window (250 bars)
Rolling batch structure. Model variants are tuned on the window ending at each batch origin; the chosen champion is deployed forward for the next 25 OOS steps before re-tuning.

Default parameters

History pool1 000 bars
Tune window250 bars (last before batch origin)
Inner train200 bars
Inner test50 bars
OOS batch size25 bars
Approx. batches~30 per 1 000-row sample
OOS predictions~750 sequential one-step predictions per run
Expanding poolYes — history grows as the window advances
Sticky championYes — prior champion retained if inner-test score ≥ challenger
PCA components5
Why rolling instead of a fixed holdout? A single train/test split is fragile: performance depends heavily on which years land in which partition. Rolling OOS averages over many regimes, is harder to inadvertently overfit to, and provides a sequential prediction record that better resembles live deployment.

5. Model selection grid

Three model families are evaluated at each tune step under the default faster grid. Larger grids (fast, all) include expanded hyperparameters and optional XGBoost/LightGBM.

ModelKey hyperparameters (faster)Rationale
Logistic Regression C=1.0 Strong baseline; calibrated probabilities; interpretable coefficients
Histogram Gradient Boosting max_depth=4, max_iter=80, lr=0.1 Non-linear, fast on small panels, handles mixed-scale features well
Random Forest max_depth=8, n_estimators=80 Ensemble diversity; robust to outliers; natural feature importance

Selection criterion: highest accuracy on the 50-bar inner test. The sticky-champion rule prevents unnecessary model churn: if the previous batch's winner ties or beats all challengers on the new inner test, it is retained. Parallel tuning (n_jobs=−1) is used throughout.

6. Sweep dimensions & artifacts

The full batch export sweep crosses four dimensions, producing a comprehensive OOS record across pairs, forecast horizons, target types, and return bases.

Pairs10 (default universe)
Leads1 – 10 bars ahead
Target modesbinary, ternary
Return basesclose-to-close, high–low
Total runs10 × 10 × 2 × 2 = 400

Output artifacts per sweep

  • run_summary — one row per run: OOS accuracy, lead, target, pair, chosen model
  • all_batches — per-batch model selection and inner-test scores
  • all_steps — per-step OOS predictions, correctness, and forward returns
  • per_run_model_stability — model family switching frequency per run
  • model_stability_global — cross-run champion distribution
  • PDF summary — headline results and aggregated stability charts

7. Intraday extension (H1 / H4)

The same pipeline is extended to hourly (H1) and 4-hour (H4) bars. The design intentionally mirrors the daily setup: same PCA pipeline, same OOS model selection logic, same export artifacts. Key differences are the lookback and lead ranges, calibrated to the higher bar frequency.

TimeframeLookbackLeadsPCA k
Daily1 000 bars1 – 105
H42 500 bars1 – 85
H13 000 bars1 – 125

Intraday bars are sourced from cached FXCM data with the same macro/technical feature merge applied to the intraday timestamps. Cross-pair context is joined at the same bar resolution.

The H4 pipeline feeds a stop-loss overlay analysis that evaluates risk-adjusted OOS returns across hold periods, providing a bridge between directional accuracy and practical trade management.

8. Daily forecast report

The daily report pipeline filters sweep results through a quality gate before generating live predictions:

  1. Select only binary close-basis runs with no errors.
  2. Apply minimum OOS accuracy threshold (default ≥ 60%).
  3. Optional endorsed-run filter from a reviewed qualified_models.json.
  4. For each retained run: refit the latest chosen model on the trailing 1 000 bars, predict the newest eligible bar.
  5. Apply NY 5 PM session calendar rule to determine the correct data-through date.
  6. Output signal, direction probability, and confidence to the daily workbook.

This keeps the production signal set conservative: only models that have demonstrated sustained OOS accuracy are included in daily forecasts.

9. Limitations & open questions

  • Transaction costs: the core backtest measures raw directional accuracy. Bid-ask spread and execution slippage are modelled separately in the stop-loss overlay but not embedded in the main OOS metric.
  • Regime non-stationarity: rolling retraining mitigates but does not eliminate the risk that calibrated feature relationships break down in novel macro regimes (e.g. post-2022 rate cycle). Model stability tables help monitor this.
  • Cross-pair correlation: pairs share macro features, so individual run OOS accuracy scores are not fully independent. Portfolio-level diversification calculations should account for this.
  • Binary framing: predicting direction, not magnitude. A correct directional call on a 2-pip move and a 200-pip move are treated equally by the accuracy metric. The forward_r columns allow magnitude-weighted evaluation separately.
  • k selection (PCA): k=5 is empirically validated but a fixed global default. The variance sweep (pca_variance_analysis.py) supports per-pair, per-timeframe tuning if needed.