Methodology & validation

Factor Watch — Methodology

How every number is computed. This file is the spec; factors.py and analytics.py implement it. If the code changes, change this file in the same commit. Transparency is a product feature: Barra and the S&P factor indices are black boxes, this is not.

1. Universe

S&P 500 constituents, point-in-time. Membership at any past date is reconstructed from FMP's historical add/remove event feed by walking events backward from the current 503-name list. A stock acquired in January 2026 is therefore in the universe for rebalances before its removal date and out afterward — no survivorship bias from using today's list, to the extent the event feed is accurate.

Secondary share classes (GOOG, FOX, NWS, and — in the 2020–2022 stretch of the backfilled era — DISCK and UA) are dropped: FMP reports the full company market cap on both lines, so keeping both double-counts the company (Alphabet was 2x weighted before this fix). The primary line (GOOGL, FOXA, NWSA, DISCA, UAA) carries the company.

History depth. The series is backfilled to the 2019-12-31 first rebalance (returns from January 2020) so the era ruler in §5 covers all of "modern markets", COVID crash included. Ticker renames whose history FMP re-homes under the new symbol are mapped in ingest.RENAMES (FB→META); ingest.survivorship_audit checks every ever-member's price coverage against its actual membership window and reports holes to the run log and data/raw/meta.json, so a recycled ticker can never silently vanish from past portfolios.

Known limits:

• Ticker renames sometimes appear as add/remove pairs in the feed (noise, small; the audit above catches the harmful cases).
• A handful of removed/delisted tickers have no usable price history on FMP (e.g. INFO — IHS Markit — whose ticker was recycled by a later listing); they drop out of past portfolios at formation, and the survivorship audit documents each one.

2. Data

All from FMP (Premium tier), pulled by ingest.py:

The FW 3000 universe builder (universe.py, §13) additionally uses company-screener, delisted-companies, historical-market-capitalization, and sp500-constituent.

Point-in-time rule: at a rebalance date t, a quarterly report is visible only if its filingDate ≤ t. No look-ahead into not-yet-filed quarters. TTM aggregates use the last 4 visible quarters.

Market cap at t: last reported quarter's market cap, drifted to t by the adjusted-price ratio. (Drifting with total-return prices slightly overstates cap for high-yield names between reports; immaterial for quintile ranking.)

Data-hygiene guards (2026-07). Vendor price errors in the broad universe's small end produced two published artifacts — an unadjusted ~160x print (ESPR, 2026-07-08) that moved five FW 3000 spread series by ±27–33% in one day, and a block of stale micro-cap tapes snapping back at once (2025-12-03, broad size spread +14%). Three guards now apply, all in factors.py:

• Split splice: a single print moving more than 8x against its neighbor is treated as an unadjusted corporate action or vendor error; the earlier price segment is rescaled so the step disappears (what a correct split adjustment would have done — within-segment returns are unchanged, the jump day goes flat). Real one-day moves beyond 8x are essentially unattested; the rare true collapse past −87.5% is knowingly flattened, costing an equal-weighted leg ~0.2% versus the ~30% a single unadjusted print injects. Every splice is logged.
• Stale tapes score no vol/beta: a name whose last 15 closes are identical is repeating a vendor placeholder, not trading; its fake zero volatility/beta would stack the low-vol book and the short beta leg, so it is excluded from those two scores at that rebalance (other factors are unaffected). The eventual catch-up print below the splice bound can still land as a one-day move — attenuated, no longer concentrated in the vol/beta books.
• Big-day attribution: any book moving more than 10% in a day logs its top single-name contributors in the run log, so the next bad print names itself instead of surfacing as an unexplained spike.

3. Factor definitions

Scores are computed cross-sectionally over the point-in-time universe at each rebalance. Raw metrics are winsorized at 2.5%/97.5%, then z-scored. Composites average the available component z-scores (a name missing one component is scored on the rest; missing all → excluded from that factor's portfolios).

4. Portfolio construction

At each month-end rebalance (last trading day):

1. Rank scored names; split into quintiles (~100 names each in the S&P 500, ~570 in the FW 3000). 2. Long-only factor series (<factor>_long): top quintile, cap-weighted with a 5% single-name cap (excess redistributed pro-rata, S&P-style). Without the cap, the quality and momentum longs degenerate into a mega-tech beta bet (6 names ≈ 60% of weight) and stop measuring the factor — this showed up directly in validation. This is the comparable to the S&P factor indices / factor ETFs and feeds the quilt. 3. Spread series (<factor>_spread): top quintile minus bottom quintile, both equal-weighted (academic convention; cap-weighted spreads are dominated by megacaps). 4. Benchmark (bench): full universe, cap-weighted. 5. Sectors (sector_*): cap-weighted within each GICS sector (current-constituent mapping; sector history not reconstructed).

Between rebalances portfolios are buy-and-hold: weights drift with prices; no daily re-ranking (daily re-ranking is the classic way to fake a factor series and leak turnover-free alpha). A name whose prices stop mid-month (delisting/acquisition) is frozen at its last price — economically a cash-out reinvested pro-rata at next rebalance.

Rebalances start at the first month-end on/after 2019-12-31, so daily series begin January 2020 — the era backfill of 2026-07 (previously 2024-06-30). The start needs a 12m momentum lookback, so price history is pulled from 2018-12-01. Note the backfill also deepened the quarterly fundamentals pull (20→44 quarters), which lets more names clear the earnings-variability minimum at recent rebalances too — recent quality quintiles can differ marginally from the pre-backfill series.

4a. Published constituent books (factor_books.json, Factors page)

The books are published, not just their returns. At every rebalance the membership captured during series construction (never recomputed after the fact, so the lists are exactly the portfolios behind the series) feeds data/derived/factor_books.json:

• Current books: every top-quintile name with its long-index cap weight, and every bottom-quintile name (the spread's equal-weighted short leg), with the rebalance score. A null weight marks a name that scored into the quintile but had no point-in-time market cap — it is carried in the equal-weighted spread legs but absent from the cap-weighted long index.
• Tenure (since): the start of the name's current unbroken run in the book, counted over computed rebalances. A skipped book (see the §4 gap warnings) neither breaks nor extends runs — diffs span to the next computed rebalance.
• Changes & turnover: adds/drops between consecutive computed rebalances of the long book; turnover is adds ÷ book size (share of the book that is new). The site shows the last 12 rebalances; the full history stays in the artifact.
• Names and sectors are today's mapping (same compromise as the sector series, §4 point 5): historical adds/drops display current metadata, and delisted names may show no sector.
• Per-name 20d/YTD returns are descriptive convenience stats on adjusted closes, not portfolio attribution.

5. Spread monitor (z-scores)

For each factor, two relative series:

• rel: long-only minus benchmark (what an ETF-vs-SPY watcher sees)
• spread: the Q5−Q1 series (the cleaner factor signal)

Factor and sector relative returns compound the **daily arithmetic return difference**: product(1 + portfolio_daily_return − benchmark_daily_return) − 1. The overview chart, table and factor detail page use this same series and the same period boundaries. This differs from the geometric ratio of compounded growth used for the thematic baskets (§14).

For horizons 1d / 5d / 20d / 60d: current h-day compounded return compared to the trailing 252 overlapping h-day returns of the same series (current excluded). Report z-score and percentile (returns are fat-tailed; the percentile keeps the z honest). |z| ≥ 2 is the headline flag.

Era context (the modern-markets ruler). Once a series has ≥750 overlapping observations (~3 years), each horizon also reports how the current move ranks against the series' entire history — the S&P series is backfilled to January 2020, COVID crash included, so this answers "how big is this move by modern-markets standards" on a near-fixed baseline. The block carries the empirical rank (nth_worst / nth_best, with the dates of the nearest prior worse/better move), the empirical percentile, and a fixed-ruler z. The empirical rank leads in all copy — with fat tails the statement "2nd-worst week since 2020" is honest where "−5σ" merely gestures. The trailing-year z remains the flagging engine: it answers "unusual lately", adapts to volatility regimes within a year, and is deliberately not replaced by the era ruler.

Caveat by design: with overlapping windows the baseline observations are autocorrelated — fine for "is this unusual?", not a t-test. A second caveat, discovered live in the June–July 2026 momentum unwind: after a volatility-regime shift, the trailing-year σ is calibrated on the calmer prior regime, so repeated 3σ+ prints can describe ONE persistent episode. The era rank and the percentile are the antidotes; read them together.

6. Rotation

Leadership state on trailing-20d returns of the long-only series: the current leader (top factor by trailing 20d return), how many consecutive sessions it has led, the previous (different) leader, and the full trailing-20d ranking. Purely informational — the panel shows the state and raises no alerts.

Two alert flags shipped here until August 2026 and were retired: a leadership-flip flag (new leader holding 3+ sessions after displacing an established run) and a quartile-jump flag (bottom-quartile 20 trading days ago, top-quartile now, or the reverse, held 3 sessions). With seven factors and a 20-day window, both mostly echoed big days rolling out of the trailing window rather than durable rotation — one live quartile flag confirmed on the minimum persistence, reversed to last place two sessions later, and spent its freshness window contradicting the ranking bars beside it.

7. Performance quilt

Calendar-month total returns of the long-only factor series + benchmark, ranked best→worst per month, trailing 13 months. Monthly granularity only — a daily quilt reshuffles too much to read; daily action lives in the spread monitor.

8. EPS revisions (V2, collection running now)

collect_revisions.py snapshots consensus EPS/revenue (avg/high/low, analyst counts) for all constituents daily. **This series cannot be backfilled — non-Enterprise vendors don't sell daily revision history — which is exactly why it compounds into a moat.** Snapshots are committed to git because they are irreplaceable. Planned outputs once ≥1 month accumulates: net up/down revision breadth across the index; revision leaders-vs-laggards spread (the Counterpoint-style edge).

9. Factor seasonality

Our own series is too short for seasonality — 2 years gives n=2 per calendar month, which is astrology. So seasonality.py uses the Ken French library monthly factor returns (momentum to 1927, HML/SMB/RMW to 1963), free: mean/median return and hit rate per calendar month, full history and trailing 30y. The total US market (Mkt-RF + RF; CRSP all-US, labeled "US market", not strictly the S&P 500) is included as a reference row. The dashboard renders the full factor × month grid as a mosaic; the digest uses the current month's baselines. The snapshot carries the current month's baseline next to the live monitor ("June is historically momentum's strongest month, 70% hit rate — and it's currently -2.4σ"). Definition mismatch (French universe ≠ S&P 500 quintiles) is acceptable for a seasonal-baseline view and disclosed in the payload. Mapping: momentum→UMD, value→HML, size→SMB, quality→RMW(profitability); low vol / dividend yield / high beta have no French analogue and are omitted.

10. Validation

validate.py checks the computed series against two independent references (report: data/derived/validation.md): 1. Published S&P 500 factor index monthly returns (Invesco dashboard quilt, Bloomberg-sourced, May 2025–Apr 2026): per-factor correlation, sign agreement, mean abs difference; per-month quilt rank correlation. 2. Factor ETFs (SPMO, RPV, SPHQ, RSP, SPLV, SPHD, SPHB vs SPY): daily return correlation, absolute and benchmark-relative.

Exact agreement is not expected (S&P indices are ~100-name score-weighted baskets; RPV is style-weighted "pure value"). What must hold: high correlation, consistent sign, same ordering most months. If a change to factors.py moves these checks materially, that's a regression until explained.

Current status (2026-06): monthly corr vs published 0.90-1.00 across all factors, MAD 0.6-2.0pp; daily corr vs ETFs 0.87-1.00. Weakest link is quality's benchmark-relative correlation vs SPHQ (~0.5).

Tested and rejected (2026-06): S&P-style accruals in quality. Swapping earnings variability for NI−OCF/assets accruals regressed every quality check (monthly corr 0.91→0.76, relative corr vs SPHQ 0.51→0.02); a 4-component mix was also worse (0.79). Probable cause: NI−OCF accruals are meaningless for financials (~15% of the universe) and FMP quarterly OCF is noisy. The accruals metric is still computed in factors.py for future experiments (e.g. excluding financials) but is not in the composite. Methodology changes must beat the validation harness to ship.

11. Breadth

factors.py also writes data/derived/breadth_daily.csv: each day, the share of point-in-time index members trading above their own 50-day and 200-day moving averages (equal count, secondary share classes excluded, names without enough history for the MA dropped from that day's denominator). Used as a participation check on factor moves — a factor rally with collapsing breadth is a different animal from a broad one.

Validation spot checks: validate.py compares every factor against the published indices (monthly) and its ETF analogue (daily) on each close. The references differ in construction by design, so they are spot checks, not anchors: a monthly correlation below 0.75 or daily ETF correlation below 0.80 — comfortably under the current 0.90–1.00 / 0.87–1.00 levels — is flagged prominently at the top of this page's validation report and in the run log, but never blocks the daily run.

Known measurement compromises (audited 2026-07): dividend yield's numerator is the split-adjusted TTM dividend while its denominator is the dividend-AND-split-adjusted price, so historical yields are inflated in proportion to dividends paid after the measurement date (decaying to zero at the latest session); market caps drifted with an adjusted-price ratio overstate caps by the yield accrued over the (≤1 quarter) drift window. Both distortions are ~monotone in true yield, so rankings shift only at quintile margins. Chart series carry cumulative levels flat across any missing sessions rather than interpolating; validate.py's internal consistency guards flag any such gap. A quilt month built from fewer than 15 sessions is marked with *.

12. Thematic baskets

Definitions live in baskets/*.json, one file per basket: thesis, members (each with an added date and a written rationale, plus a removed date when dropped), and a changelog. Membership is curated by the maintaining agent; **every add/drop is dated and justified in the changelog — the audit trail is part of the product.**

Construction (baskets.py): equal-weighted across active members, buy-and-hold between rebalances — the same discipline as the factor portfolios (no daily re-ranking, frozen prices on delisting). Rebalances fall at month-ends and on any dated membership change, so adds and drops take effect on their changelog date with an equal-weight reset rather than waiting for the next month-end. Members without price data are excluded from the series and surfaced in the dashboard payload for review; names that listed mid-history simply enter the series when their price tape begins. Series before a basket's creation date are a **backtest of the membership as of creation**; live tracking starts at creation. Benchmark-relative figures compound the basket and the computed cap-weighted benchmark over the same window.

ETF allocations: baskets with weighting: "target" carry an explicit weights map of positive weights summing to 100%. The 60/40 SMH/XLE basket holds 60% SMH and 40% XLE at each month-end reset; weights drift between resets. It uses the ETFs' dividend- and split-adjusted price histories and the same buy-and-hold return calculation. Both ETF legs must have valid prices throughout each holding period; incomplete allocations fail the build instead of silently reallocating to the available leg. The holdings are identical in FW 3000 and S&P 500 views; only the computed comparison benchmark changes. ETFs are not flagged as stocks that left the selected index. History before September 9, 2026 is a backtest.

Membership aims for a **wide cross-section of each theme inside the S&P 500** rather than a concentrated best-ideas list: every leg of the theme with a listed index constituent should be represented. Some themes are structurally thin in a large-cap universe (cybersecurity's pure plays mostly trade outside the index); their changelogs say so rather than padding the basket with weak fits. Members that leave the S&P 500 are flagged by the pipeline (ex_index in the payload, surfaced on the dashboard) and removed with a dated changelog entry.

Two universes. Each basket is computed over both the S&P 500 and the FW 3000 (§13). Members tagged "universe": "broad" in the basket definition exist only in the FW 3000 version — off-index pure plays like ZS or OKTA in cybersecurity, mid-cap banks in regionals — while untagged members count in both (every S&P 500 name is an FW 3000 name by construction). One file, one changelog, one thesis per theme; names that leave the S&P 500 but remain in the FW 3000 move to a broad entry.

Basket risk statistics

Each basket detail page reports Sharpe, Sortino and historical volatility from its committed, unrounded daily total returns, in the selected universe (basket_pages.py). Risk has its own Period control, independent of the performance chart: 5d, 20d, 60d, MTD, YTD and 1y; the default is 1y. MTD/YTD include daily returns in the snapshot's calendar month/year, while 1y contains exactly 252 trading sessions. The page shows the actual dates, valid observations and requested session count.

If a newly published basket has no column in the stored daily return file, its adjacent published chart levels supply estimated daily returns. Such readings are explicitly marked ≈ and labeled as estimates from published chart levels; rounding can affect the result. This fallback does not replace missing values in an existing return column. The risk block carries its own observation date when the live price edition is newer than stored return history.

For daily simple returns R, with N observations:

• Sharpe ratio: sqrt(252) × mean(R) / sample_std(R). The sample standard deviation uses N − 1. The risk-free rate is explicitly assumed to be 0%, since no cash-return series is supplied; this is not a ratio of excess returns over observed Treasury or cash returns. The formula follows the historical mean/standard-deviation definition in William Sharpe's The Sharpe Ratio.
• Sortino ratio: sqrt(252) × mean(R) / sqrt(mean(min(R, 0)²)). The downside target is 0%. The denominator averages over all N sessions, treating nonnegative returns as zero shortfall, rather than averaging only losing sessions. This uses the target downside-deviation convention described in CME's Sortino paper.
• Historical volatility: sqrt(252 × mean(ln(1 + R)²)), shown as a percentage. This matches the zero-mean log-return realized volatility on the Vol page (§15), not the demeaned simple-return standard deviation in the Sharpe denominator. Ratio numerators use the arithmetic daily mean, not the compounded return or CAGR.

All three statistics require the full selected period and at least **5 sessions**. The benchmark's dated observations define the session grid; returns after the basket snapshot date are excluded. Missing basket dates, blank/nonfinite returns, or returns at or below −100% invalidate the window. The window is never stretched, filled with zeros or silently shortened. Calendar windows also require benchmark history before the calendar boundary, and the grid must reach the snapshot date. Unavailable readings show — and a reason. Sharpe is undefined when the daily standard deviation is at most 1e-12; Sortino is undefined when downside deviation is at most 1e-12, including a window without losses. A genuinely flat series has 0.0% historical volatility, with undefined ratios.

Square-root-of-time annualization is a reporting convention, not a forecast; serial dependence can limit its interpretation. Short-window ratios can be unstable. These statistics inherit the basket's pre-creation backtest and intraday partial-session caveats, include dividends and use total basket returns even when the performance chart displays benchmark-relative returns.

Reference-ETF cross-check. Where a listed sector/industry ETF — or a simple blend, e.g. 50/50 XLP+XLU for the defensives basket — approximates a basket's theme, the basket definition pins it as a reference, and baskets.py reports the daily return correlation against it: absolute, and benchmark-relative (basket minus our computed benchmark vs ETF minus SPY) so agreement is not just shared market beta. The numbers render on each basket's dashboard section in both universes; the S&P versions also appear in the validation report. References are sanity checks, not tracking targets: they are mostly cap-weighted, hold names outside the index, and follow different rules, so correlations should be high but not 1.0. A persistent drop is a prompt to review membership — deliberately not an automated failure, unlike the factor guardrails in §10.

13. FW 3000: the broad-universe mirror

The computation chain — ingest, factor construction, baskets, and the analytics snapshot — runs twice: once over the S&P 500 (the default views) and once over the FW 3000, our self-built Russell 3000-style broad universe, selected with the universe switch on the Overview and Baskets pages. Same factor definitions, same quintile construction, same z-scoring — only the universe changes. The revisions collection (§8), seasonality baselines (§9), the published-index factor validation (§10), and the daily digest run once, on the S&P side; the FW 3000 carries its own universe-level guardrail (below).

Construction (universe.py): the largest 3,000 actively trading US common stocks by market capitalization (NYSE/NASDAQ/AMEX, price > $1, one share class per company with known secondary classes deprioritized explicitly; preferred shares, warrants, rights, units, funds, and exchange notes excluded), always including every current S&P 500 member (some index members have foreign headquarters or fresh ticker renames that screeners mishandle). Reconstitutions are at calendar quarter ends, plus a provisional segment maintained by the weekly rebuild: the membership cut since the last quarter end is re-cut each Monday and only becomes final at the next quarter end. Known limit: history candidates are today's largest ~4,500 actives plus names delisted since mid-2024, so a still-listed stock that fell far out of today's top ~4,500 can be missing from historical quarters it belonged to — a mild winner bias the survivorship stats below do not capture. FMP carries no licensed Russell membership, so we build our own the way Bloomberg builds its B3000 — this is **not the Russell 3000**, and we never label it as such; it is a Russell 3000-style universe under the published rules in this section. Membership history back to mid-2024 is reconstructed point-in-time from historical market caps, including delisted names.

Survivorship is measured, not assumed away. At the first build, 98.4% of the 1,779 names delisted since mid-2024 had enough market-cap history to be ranked, 675 of them entered the reconstructed membership, and 17.7% of the June-2024 membership is no longer trading — those names stay in the series for the quarters they belonged. The numbers live in data/universe/fw3000_meta.json and refresh with each weekly universe rebuild.

Validation: the FW 3000 cap-weighted benchmark is checked against IWV (iShares Russell 3000 ETF) on every close run; a daily correlation below 0.97 raises a prominent spot-check warning (it does not block the run — one transient bad print once cost a full close). First build: correlation 0.997 over the full series, ~1.3% annualized tracking error, cumulative return within ~0.6pp of IWV over the backtest window. Seasonal baselines come from the Ken French library (§9), whose all-exchange universe is much closer to the FW 3000 than to the S&P 500.

Cadence: the FW 3000 refreshes on the same schedule as the S&P 500 — every 30 minutes through the session from 9:35 ET (prices only, off the cached fundamentals) and again at the close. Fundamentals refresh on close runs (Mondays and the month-end window, so rebalance inputs are fresh); the universe membership rebuilds on Mondays. Both universes therefore carry the same "as of" date through the day, and the intraday/close marker on each page distinguishes a partial session from a finalized close. The broad intraday refresh is best-effort: if it is ever unavailable, the FW 3000 pages fall back to the last committed close rather than block the S&P update.

14. Estimate revisions

Factor Watch snapshots the analyst FY consensus (EPS and revenue: mean, high, low, analyst count) once per market day (collect_revisions.py): every S&P 500 constituent since 2026-06-10, widened to the full FW 3000 membership on 2026-07-04. Small caps without analyst coverage store no rows — that absence is itself the coverage truth. The S&P series is era-aware: broad-era snapshots are intersected with the committed S&P membership so its trend never silently changes universe; the FW 3000 series publishes once it has six broad snapshots. Published history of when consensus moved cannot be bought later — the archive only exists because it is collected daily, which is why the site treats it as its most defensible dataset.

Net revision breadth (site section "Estimate revisions", sp500): for each constituent, compare today's FY1 EPS consensus to the reading 5 sessions earlier, matched on the same fiscal year (a fiscal-year roll is never counted as a revision). A name counts as raised/lowered when the consensus moved by at least 0.1% relative; names covered by fewer than 3 analysts are excluded. Net breadth = (% raised − % lowered).

Factor-quintile cut: the same breadth measured inside each factor's top and bottom quintile, using the committed membership from the latest monthly rebalance (latest_portfolios.json). A positive top-minus- bottom spread reads as fundamental confirmation of the factor; a negative one as analysts leaning against it.

Measurement notes: consensus levels come from the vendor's estimate aggregation and can jump when analysts are added/dropped, not only when targets change; the 0.1% threshold and the analyst-count floor damp but don't eliminate this. The series' trailing depth is bounded by the collection start date above.

15. Realized volatility

The Vol page tracks realized (historical) volatility for every daily return series the pipeline computes — the cap-weighted benchmark, the long-only factor portfolios, the Q5−Q1 spreads, the GICS sectors, and the thematic baskets, in both universes. Everything derives from the committed return series above; no new data enters the calculation (vol.py, output data/derived/vol.json).

Definition. Realized vol over a w-session window is the zero-mean volatility of daily log returns, annualized: RV_w = sqrt(252 × mean(r²)) over the last w sessions, with r = ln(1 + daily return). No demeaning — over short windows the sample mean is noise and costs a degree of freedom, and the sum-of-squares convention matches the realized-variance literature. Windows: 5, 10, 20, 60, 120, 252 sessions. Interior data gaps poison their windows (no vol is reported across a gap) rather than being interpolated.

The "1d" reading. One daily close cannot yield a standard deviation, so the 1d figure is the session's absolute log return restated in annualized vol units, |r| × sqrt(252) — "the tape moved like a 33-vol market today" — ranked against the trailing year of absolute daily moves. The visible 1d return column shows the signed daily simple return. Its color and sort order use the absolute-magnitude percentile; the tooltip shows the annualized magnitude described above. Neither is a multi-session volatility estimate.

Percentiles and z. Each window's current reading is ranked against the trailing 252 daily readings of the same window (current excluded), the same ruler as the spread monitor (§5), with a z alongside. Once a series has ≥750 readings (~3 years), an era block ranks the reading against the full history since 2020 ("3rd-highest 20d vol since 2020", with the date it was last exceeded). Same fat-tails discipline as §5: the empirical rank leads, the z rides along. Overlapping windows make neighboring readings autocorrelated — the percentile answers "is this unusual?", never a t-test.

Derived reads.

• EWMA vol: the RiskMetrics recursion var_t = λ·var_(t−1) + (1−λ)·r_t², λ = 0.94, annualized — a faster-reacting companion to the flat windows.
• Δ20d: 20-day vol now minus 20 sessions ago, in vol points — the expansion/compression tape.
• 5−60 curve: 5-day minus 60-day vol, in vol points. Persistently positive = short-horizon vol above long, the realized-vol signature of a stress regime.
• Vol cone: per window (5/10/20/60/120), the 5th/25th/50th/75th/95th percentiles plus min and max of every overlapping reading in the full history, drawn behind the current term structure. Published once a window has ≥250 readings, with n disclosed.

Known limits (by design). Close-to-close only: the series are portfolio composites with no OHLC tape, so range-based estimators (Parkinson, Garman–Klass, Yang–Zhang) don't apply, and intraday paths are invisible beyond their close prints. Spread vol is the vol of an equal-weighted Q5−Q1 return difference — a property of the factor, not of a tradable cost-free strategy. Basket series before a basket's creation date are backtests (§12), and their vol history inherits that caveat. On intraday refreshes the last "day" is a partial session, so the shortest windows read slightly low until the close print lands (the page's intraday marker flags this, as everywhere on the site).

16. Basket and factor correlations

The Correlation page compares thematic baskets and primary factor **long portfolios** within the selected universe. Its default view is baskets × factors over 60 sessions; basket-only, factor-only and combined matrices use the same calculation. The overview's existing factor-correlation summary continues to use Q5−Q1 spreads and is a different comparison.

Inputs and definition. correlation_pages.py reads the committed basket_returns_daily.csv and factor_returns_daily.csv during the site build. Pearson correlation is computed from daily simple returns, not cumulative index levels. Total-return mode uses the published portfolio returns. The default benchmark-relative mode subtracts the selected universe's benchmark return from each portfolio's return on each date: corr(portfolio_A − benchmark, portfolio_B − benchmark). This measures co-movement in outperformance; it is not beta-adjusted regression residual correlation, a holdings-overlap measure, or an estimate of factor exposure.

Period and missing data. The page uses the site's period tabs: 5d, 20d, 60d, MTD, YTD and 1y. Day periods use that many benchmark sessions; 1y uses 252 sessions. MTD/YTD select benchmark return dates in the snapshot's calendar month/year, including the first session's daily return. They are not rolling approximations. A 1d option is omitted because a single daily observation cannot define a correlation.

Only sessions with finite benchmark returns on or before the snapshot date are eligible. Periods are not extended to compensate for missing observations. Each pair uses matching dates with finite returns in both series. For a period of N sessions, require max(5, min(20, N), ceil(0.8 × N)) observations: every session for periods up to 20, at least 20 and 80% coverage for longer periods, and an absolute minimum of five. Early MTD/YTD periods remain unavailable until five paired daily observations exist. For calendar periods, N is the count of benchmark sessions in that calendar period; for rolling periods it is the requested session count. Missing/nonfinite values are never zero-filled or forward-filled. A correlation is unavailable below that threshold or when either centered sum of squares is at most 1e-24 (no meaningful return variation). This also applies to the diagonal.

Every cell carries its usable observation count and first/last matching dates. Endpoints do not imply an unbroken history: internal gaps can remain. The page shows the actual window dates and labels a partial intraday observation when included. Since pairs can have different samples, the combined matrix is not guaranteed to be positive semidefinite.

Interpretation. Positive correlations indicate movement together; negative correlations indicate opposing movements; values near zero indicate little linear relationship over the selected sample. This is a description of history, not a promise of future diversification. Basket history before creation retains the backtest convention in §12. Only aggregate statistics are embedded in the page; no extra API calls or raw constituent data are required.

Brief and weekly wrap. The editorial fact pack uses these same calculations for the S&P 500 universe. It includes the highest and lowest valid 60-session benchmark-relative pairs, with their 20-session and 1y readings and 60-session total-return counterparts. Each coefficient retains its paired observation count and dates. A current reading must reach the snapshot date, and at least 60 benchmark sessions must exist; a stale basket catalog is excluded.

With at least 80 benchmark sessions, the engine also compares the current 60-session relative coefficient with the same pair's 60-session reading ending 20 sessions earlier. The largest absolute change of at least **0.30 coefficient points** is eligible for one correlation story. Both windows must meet the page's coverage rules and reach their respective end dates. They overlap by 40 benchmark sessions; this is a descriptive editorial filter, not a test of statistical significance or an independent-sample comparison. Repeated coverage of the same pair and direction within seven sessions reduces its ranking weight. The writer may use the fact pack as supporting context without making correlation the lead. Coefficients are signed numbers on −1 to +1, never percentages.

17. Chartbook

The Chartbook updates with each market-data run and contains six market charts, four economic charts, eleven sectors, seven factors, every active basket (27 at launch), and every current S&P 500 constituent security, including separate share classes, grouped by sector, industry and ticker. The full book is always available. FW 3000 / S&P 500 changes the sector, factor and basket portfolios; stock chapters always use current S&P 500 membership.

Market essentials. IEF is the iShares 7–10 Year Treasury Bond ETF using FMP dividend- and split-adjusted closes. Gold (GCUSD) and copper (HGUSD) use FMP continuous futures closes, quoted in dollars per troy ounce and per pound. Contract rolls can affect these series; no additional roll adjustment is applied. WTI is the FRED Cushing spot series DCOILWTICO, dollars per barrel. The dollar is the Fed broad trade-weighted index DTWEXBGS (January 2006=100), not DXY. Treasury yields are FRED DGS2 and DGS10, constant maturity, percent. Yield changes use basis points; other market charts show percentage changes except where a nonpositive starting price makes that interpretation invalid.

Economy. Headline/core inflation are the twelve-month percentage changes in CPIAUCNS and CPILFENS, not seasonally adjusted, joined to the same calendar month a year earlier. Real GDP growth A191RL1Q225SBEA is already quarterly, seasonally adjusted and annualized; it is not annualized a second time. Jobs added is the calendar-month difference in PAYEMS, in thousands of jobs, seasonally adjusted. Unemployment is UNRATE, seasonally adjusted, percent. GDP and jobs use bars with a zero baseline. Percentage-rate changes use percentage points; jobs changes use thousands, not investment returns.

Economic observations keep their monthly/quarterly frequency. Their dates refer to the covered period, not publication dates. The expanded chart shows the observation period, FRED's reported update time (when available), source retrieval time and next release date separately. FRED's update is not the originating agency's release timestamp. These are the latest revised histories, not vintage data suitable for an as-known-at-the-time backtest. Monthly series are not marked stale merely because their observation month precedes today. When a FRED_API_KEY repository secret is configured, the collector uses FRED's documented observations/series API; otherwise it uses public FRED CSVs. Transient requests get one bounded retry. Retained sources are labeled "Refresh delayed" and generate a workflow warning when a refresh fails.

Charts and periods. Market windows share the edition's benchmark session calendar: 20d/60d/1y mean 20/60/252 sessions with a prior-close baseline; MTD/YTD begin at the preceding month/year-end close. Custom accepts 1–600 trading sessions with the same prior-close baseline, limited to the available edition history. Economics use separate 1/5/10 calendar-year windows, default 5y. The latest plotted observation and first available observation determine each market period change; the latest daily change is also shown. Economic changes compare the latest and previous native-frequency observations and do not change with the viewing window. A short history is labeled; no value is extended to a day without an observation. Each card has its own linear vertical scale and units. Portfolio levels rebase to 100 at the first displayed observation. Sectors are cap-weighted (§4); basket histories retain §12's backtest convention. Individual security prices are adjusted historical closes, with the latest provisional price on intraday runs. Intraday stock prices reuse the pipeline's timestamp-validated price cache (§8); they are periodic snapshots, not a streaming feed. Today's partial-session portfolio returns use the same calculations as the overview. Intraday observations are labeled provisional on cards, expanded charts, image exports and printouts.

Factors offer Long only and Long/Short views in either universe. Long only compounds the published top-quintile <factor>_long total returns, cap-weighted with a 5% single-name cap (§4). Long/Short compounds the published <factor>_spread daily returns: equal-weighted Q5 minus equal-weighted Q1. It does not subtract the short leg from the cap-weighted Long only series. Both views are indexed to 100 at the first displayed observation, and their daily and period changes are percentage returns on the selected compounded index. The selected basis also applies to expanded charts, moving averages, copied images and print output.

Optional daily-series SMAs average the most recent 50 or 200 **available observations**, including pre-window history. A missing observation remains a gap in both the price line and the average; it is not zero-filled or counted as a new observation. No moving averages are applied to economic panels. Portfolio wealth histories use the last contiguous run of available returns rather than compounding across an unknown return. The source retains 600 daily records for the display window and moving-average warmup.

Refresh and export. chartbook_data.py runs after every portfolio calculation, using --intraday for provisional sessions. It validates portfolio histories against their snapshots, matching basket dates, and current stock coverage, and writes all content-addressed chapters before atomically replacing the edition manifest. FRED retrieval failure retains that series' last observations and retrieval timestamp; successful responses are reused for at most 15 minutes so morning releases can appear during the day. A macro-provider failure retains dated macro charts without blocking stock and portfolio updates. IEF, gold and copper accept positive, timestamped quotes from the current New York session on intraday runs; otherwise they retain their dated observations. Treasury yields, the dollar, oil and economic series retain the publisher's observation dates and schedules. These are never relabeled as today's data. The S&P session sets the chart calendar; if the best-effort broad update is behind, its portfolio charts retain and label their older observations. Closing editions still require matching universe dates. A failed equity export fails the data run instead of silently deploying an older book. Source-only builds remain offline, using the collected edition, and reject a book whose recorded portfolio snapshots no longer match their inputs. Previous-edition chapter files remain available for readers with an older open tab. Older tabs can reload if a chapter is no longer available.

Copy image includes the chart's period, universe where applicable and units. Print / PDF loads every selected chapter, including unvisited stock chapters, and renders four charts per landscape Letter page with edition and page labels. Missing chapters produce an explicit unavailable panel. Native browser print without preparation uses loaded chapters and identifies unprepared chapters.

Sources: FRED series, FMP adjusted historical prices, FMP commodity histories.

Differences vs the big-shop dashboards (deliberate)

• Quintile spreads instead of score-weighted index baskets: simpler, fully reproducible, and symmetric (you see the short leg).
• Equal-weighted spread legs: standard in the literature, less megacap contamination.
• Composite definitions are minimal (3 components max) and listed above in full. No proprietary descriptors, no opaque "crowding" overlays.

18. Exchange calendar and brief corrections

Next-session dates use the committed NYSE equity calendar in config/nyse_calendar.json, verified against the exchange’s published calendar. Coverage is explicit (currently 2026–2028). Full closures and early closes are separate: an early close is still a trading session. New Year’s Day 2028 falls on Saturday and does not close December 31, 2027.

market_calendar.py returns an unknown result outside verified coverage or when the cache cannot be read. A missing calendar is distinct from a valid window with no holidays. The brief then uses “Looking ahead” without a confident next-session date. Other market-data failures do not remove the independently available exchange calendar. Review this cache when the exchange publishes new dates or an exceptional closure.

Emailed brief records remain unchanged. Calendar corrections on dated web pages carry a dated notice with both the original and corrected values; they do not silently rewrite an emailed observation.

Disclaimer

Factor Watch is provided for informational and educational purposes only. Nothing on this site, in the email brief, or in any other output of this project is investment advice, a recommendation, or an offer or solicitation to buy or sell any security.

The pipeline is automated and depends on third-party data sources. Everything here is provided "as is", without warranty of any kind: no guarantee that any figure is accurate, complete, or timely, or that the site and pipeline are free of errors or bugs. Data may be delayed, revised, or simply wrong; runs can fail or publish incorrect output. Verify independently before relying on anything shown here.

We accept no liability for any loss or damage arising from the use of this site or its data. Consult a qualified financial adviser before making investment decisions.

Factor Watch validation report

Computed series: 2020-01-02 to 2026-09-11. References: S&P 500 factor index monthly returns (Invesco dashboard quilt, as of 2026-04-30) and factor ETF total returns (FMP, dividend-adjusted).

momentum (ETF ref: SPMO)

Monthly corr vs published: 0.90 | sign agreement: 83% | mean abs diff: 2.1pp

value (ETF ref: RPV)

Monthly corr vs published: 0.92 | sign agreement: 100% | mean abs diff: 1.1pp

quality (ETF ref: SPHQ)

Monthly corr vs published: 0.91 | sign agreement: 92% | mean abs diff: 1.1pp

size (ETF ref: RSP)

Monthly corr vs published: 0.92 | sign agreement: 92% | mean abs diff: 1.1pp

lowvol (ETF ref: SPLV)

Monthly corr vs published: 0.98 | sign agreement: 92% | mean abs diff: 0.6pp

divyield (ETF ref: SPHD)

Monthly corr vs published: 0.96 | sign agreement: 83% | mean abs diff: 1.1pp

bench (ETF ref: SPY)

Monthly corr vs published: 1.00 | sign agreement: 100% | mean abs diff: 0.1pp

Daily return correlation vs ETF analogue

Sector series vs SPDR sector ETFs (daily corr, last 252 trading days)

Quilt rank agreement (Spearman, ours vs published, per month)

Thematic baskets vs reference ETFs

Each basket that has a listed sector/industry analogue declares it as a reference in its definition; baskets.py reports the daily return correlation, absolute and benchmark-relative (basket − our benchmark vs ETF − SPY). References differ in universe and weighting, so high — not perfect — agreement is the expectation. Informational, no guardrail.

FW 3000 benchmark vs IWV (iShares Russell 3000 ETF)

Daily correlation 0.9949 over 1669 sessions; annualized tracking error 2.10%; cumulative +139.5% (FW 3000) vs +149.3% (IWV). Spot-check floor: 0.97.

Internal consistency (gap and corrupted-book guards)

All factor series continuous, no outlier months. Clear.