# Data notes This is assigned reading, not boilerplate. Every backtest you run in Portfolio Lab is a statement about history, and the data underneath it has known shape and known holes. If you do not know what they are, you will read a result as stronger than it is. Runbook reference: section 4. Formula definitions live in [METRICS.md](METRICS.md). --- ## 1. Where the data comes from | Dataset | Source | Licence | |---|---|---| | Daily OHLC, adjusted close, volume | Yahoo Finance via `yfinance` | Free, redistribution restricted. Packs go to the class LMS only, never to a public repository or host | | Daily Fama-French three factors, momentum, risk-free rate | Ken French Data Library, Dartmouth | Free, public | | Fundamentals | SEC EDGAR XBRL company facts at `data.sec.gov` | Public domain | | Listings and sector code | SEC `company_tickers.json`, `company_tickers_exchange.json`, and the submissions endpoint | Public domain | There is no FRED dependency and none may be added. The daily RF column of the French file **is** the pack's risk-free series, which is what makes the Sharpe ratio and the factor regressions internally consistent: both sit on the same risk-free rate by construction. Everything is US-listed and priced in USD. No currency conversion exists anywhere in the codebase. Canadian and other foreign issuers enter only through their US listings. --- ## 2. Survivorship bias, and what is and is not affected **The equity universe is current membership as of the build date, so it is survivorship-biased.** The universe is the 1,000 most liquid US-listed common shares ranked at build time. A company that failed, was acquired, or fell out of liquidity before the build date cannot appear in it, no matter how large it was during your backtest period. What that does to a result: it tilts every backtested return upward relative to what an investor could actually have earned, because the names that would have hurt you most are the ones most likely to be absent. It bites hardest on strategies that would have held small, distressed, or highly volatile names, which is exactly where a value screen or a low-price screen tends to point. Treat an absolute return figure from any backtest here as optimistic. Comparisons between two strategies over the same universe are much less affected, because both carry the same bias. **The fundamentals snapshots do not have this problem.** They are point-in-time: a snapshot dated D contains only facts that had actually been filed with the SEC on or before D. A figure restated later never displaces the figure as first reported. So a screen that ranks on last year's book equity is ranking on the number that was public at the time, not on the number the company later corrected it to. **Sector is current as of the build date, not point-in-time.** The SIC code comes from the registrant's current submissions record. A company that reclassified its SIC code carries its current sector in every snapshot. This is a small effect relative to the universe bias above, but it is not zero. **The ETF list is curated and static within a term.** It is not survivorship-free either: it lists funds that exist today. Funds that closed are absent. --- ## 3. Price adjustment policy Two price representations ship, and each has exactly one job. | Column | Adjusted for | Used for | |---|---|---| | `adj_close` | splits **and** dividends | All return computation and all return-based signals. Total return, always | | `open`, `high`, `low`, `close` | splits only | Trade-rule trigger evaluation, margin price classification (the $10 rule), price display | | `volume` | splits (multiplied by the split factor) | Dollar volume, so it stays consistent with split-adjusted price | **Price levels you type are in split-adjusted terms as of the pack build.** If you set a stop at $50 on a name that has since split 2-for-1, the pack's price history for that name is on the post-split basis, and $50 means $50 on that basis. This is the only convention under which a price level means the same thing across the whole history. The source ships OHLC and volume already split-adjusted, with the dividend adjustment applied only in `adj_close`. The builder does not assume that silently: it records the convention in the manifest and re-checks it on every build against every split in the window, reporting any name-day where the observed price move across a split disagrees with the assumed convention. ### Why `adj_close` is never used for market cap Market cap is **split-restated shares times the split-adjusted close**, never `adj_close`. `adj_close` is reduced by the accumulated dividend adjustment, and that reduction differs across names according to how much they have paid out. Using it would understate market cap unevenly across dividend payers, which would quietly distort every price ratio (P/E, P/B, P/S, P/CF) and every size screen. A high-yield utility would look systematically smaller and cheaper than an identical company that paid nothing. --- ## 4. The share restatement, with a worked example Shares outstanding are stored **split-restated onto the price basis**, not as reported. The builder multiplies the as-reported figure by the cumulative split factor between **the date the figure itself is stated as of** and the build date, which is the same factor the OHLC series already carries. A share count is dated on the cover page or balance sheet it comes from, and that date sits before the snapshot the figure lands in, typically by weeks. Anchoring on it is what makes the correction complete. **Worked example.** Suppose a company reports 100 million shares outstanding in a filing available at the 4 January 2021 snapshot, and its stock closes at $400 that day on the as-traded basis. On 15 June 2021 it splits 4-for-1. The pack is built in 2023. | Quantity | As reported / as traded | In the pack | |---|---|---| | Close on 2021-01-04 | $400 | $100 (split-adjusted: 400 / 4) | | Shares outstanding at 2021-01-04 | 100,000,000 | 400,000,000 (restated: 100M x 4) | | Market cap at 2021-01-04 | $40bn | $40bn | The two adjustments cancel, which is the point. Without the restatement the pack would carry 100 million shares against a $100 price and report a $10bn market cap: a quarter of the truth. That company would sail through a small-cap screen it has no business passing, and every one of its price ratios would be wrong by a factor of four. Because the restatement depends on the build date, it is computed in the hygiene pass that runs before truncation, so every pack cut from one build carries the same restated figure. Two packs built months apart are two builds, and they restate independently. **A second worked example, for the case the anchor decides.** Suppose the same company's latest available share count at the 3 January 2022 snapshot is the one it stated as of 25 October 2021, and it splits 2-for-1 on 15 December 2021. That split is after the count and before the snapshot. Anchoring at the snapshot date finds no split after 3 January 2022 and stores the count unchanged, so the pack reads half the true size. Anchoring at the count's own as-of date of 25 October 2021 finds the split, doubles the count, and market cap comes out right. `SPLITCO` in the committed fixture pack carries exactly this case, alongside the first one, so both are covered by a test. **How the rule got here.** The build runbook originally specified the snapshot-date anchor, and the gap above was a stated limitation rather than a defect: the builder counted and listed every affected name in `build_report.json` so it was visible rather than invisible. The rule was amended to the as-of-date anchor on 18 August 2026, on the instructor's decision, before any pack a class runs on was built. The manifest records which anchor the build used, in `sharesRestatementAnchor`, and the affected names are still listed. --- ## 5. Two liquidity measures, deliberately different There are two, and they are not the same number. Unifying them would be a mistake. | Where | Measure | Window | Why | |---|---|---|---| | Universe construction | **Median** daily dollar volume | 252 trading days | Must be robust to one-off spikes over a long window. One earnings-day or index-add volume burst must not buy a thin name a permanent seat in the top 1,000 | | Runtime filter (`minAvgDollarVolume`) | **Mean** daily dollar volume | 63 trading days | This is the conventional definition of average daily dollar volume. A student who types 2,000,000 into a field labelled "minimum average dollar volume" means the ordinary thing by it | The 63-day runtime window is fixed and not editable. --- ## 6. Data hygiene - **Forward-fill**: a gap of at most 5 trading days is filled forward from the last observation. A gap **longer** than 5 trading days is left missing in full, not filled for its first five days. The limit is a property of the whole run of missing days. - **A filled day carries no volume.** Filling volume forward would fabricate liquidity that a minimum dollar volume filter then reads as real trading. A null volume simply drops out of the trailing mean. - **Leading and trailing gaps are never filled.** A name that has not listed yet has no price, and a name whose data ends mid-window has genuinely ended. The engine force-exits a held position at its last available close and flags it (`forcedExitDataEnd`). - **Returns exist only between actual or filled observations.** A null price yields a null return, and null returns exclude the name from signal computation on the affected windows. - **Drop rules**: a ticker whose entire fetch fails, or which has fewer than 100 observations in the full window, goes to `dropped.csv` with its reason. Late IPOs otherwise stay in the pack with short history, and their signals return null until their lookbacks are satisfied. A name with no observation on or before a release's cut is dropped from that release too, with the cut date in its reason: the minimum counts over the full fetch, and a name that listed after the cut would otherwise leave the price file silently while its fundamentals and sector stayed. Added 2026-09-06 for the first test pack, whose cut was seventeen months before its build date; a design pack cut weeks before its build date never meets the case. The pack's `dropped.csv` states the observation minimum without the count of days, "fewer than 100 observations in the fetch window", since a count over the full fetch would say how long a listing after the cut had traded; the instructor's build report keeps the count. - **One trading calendar**: the SPY trading dates inside the pack window. All series align to it. There are no cross-market calendar issues by construction. - **Hygiene runs once, on the full fetched history, before the pack is truncated.** This ordering is load-bearing. If hygiene ran after the cut, a price gap straddling the truncation date would be forward-filled in one pack and left missing in another, and the audit runner's comparison would fire on an honest submission. The guarantee holds within a build; it is not claimed across builds, because the price source restates adjusted closes and a later build refetches. Every long gap is recorded per ticker in the pack manifest under `longPriceGaps`. --- ## 7. Fundamentals: coverage, construction, and what is missing ### Point-in-time construction A snapshot dated D includes only XBRL facts with a filing date on or before D. Where a period was reported more than once, the value used is the one **first** filed on or before D. That is the only choice which is neither a restatement nor a look-ahead. Snapshots exist at **period start dates and nowhere else**. This is a deliberate structural constraint, not a data limitation: fundamentals may establish which names a strategy trades, and price and volume drive everything after that. Because no fundamental value exists on any other date, a strategy cannot rebalance on one. ### The seven base metrics Revenue TTM, net income TTM, operating cash flow TTM, book equity, total assets, shares outstanding, and sector. Each numeric metric has an ordered fallback tag list across the us-gaap and ifrs-full taxonomies; the first tag that resolves wins. **TTM construction**: `FY + YTD_current - YTD_prior` where quarterly year-to-date facts exist. Where only annual facts exist, the latest fiscal-year value whose period end is within 15 months of the snapshot date, else null. Alongside the seven, the pack ships `revenue_ttm_prior`: the same revenue tag list resolved for the twelve months ending one year before the current TTM window. It is not an eighth base metric and adds no tag work; it exists because revenue growth is defined against a year-ago TTM reconstructed from the same filing history, and only the builder holds that filing history. **Diluted EPS and total debt are deliberately excluded.** EPS adds tag work without adding a ratio, since P/E derives from market cap over net income. Debt tagging is the messiest area of XBRL in both taxonomies, so leverage is derived from assets minus equity instead. ### Fallback tags added during build The runbook's tag lists are a starting point and are extended as real coverage gaps appear. Every addition is recorded here. Each one below was made against measured coverage, not anticipated coverage, and each names the filer that exposed the gap. | Metric | Tag added | Why | |---|---|---| | Revenue TTM | `us-gaap:RevenueFromContractWithCustomerIncludingAssessedTax` | AAON and filers like it tag only the Including variant. Without it they have no revenue, and therefore no P/S, no net margin, and no revenue growth | | Net income TTM | `us-gaap:NetIncomeLossAvailableToCommonStockholdersBasic`, then `us-gaap:ProfitLoss` | Axcelis tags `NetIncomeLoss` only from 2023 onward and carries its whole earlier history under these. The order mirrors the runbook's own IFRS ordering: parent-only concepts first, the including-NCI total last | | Book equity | `us-gaap:StockholdersEquityIncludingPortionAttributableToNoncontrollingInterest` | AAON and filers like it tag only the including-NCI total. Book equity is defined as parent only, so where this tag supplies the figure the `us-gaap:MinorityInterest` balance at the same period end is subtracted back out. A filer reporting no MinorityInterest has none, and the total is already the parent figure | | Shares outstanding | `us-gaap:CommonStockSharesIssued`, `ifrs-full:NumberOfSharesOutstanding`, then `us-gaap:WeightedAverageNumberOfSharesOutstandingBasic` | `dei:EntityCommonStockSharesOutstanding` exists for 24 of 25 sampled registrants but is **current** for only 19, because the companyfacts API carries cover-page facts sparsely. ACM Research would otherwise have taken a share count dated March 2019 into a 2025 snapshot | **The weighted-average share count is a stated approximation.** It is the last tag in the list and is a period average rather than a point-in-time count. It is used only when every point-in-time tag is absent or stale, and it is the only share tag that is current for every sampled filer, so the alternative is a null rather than a better number. Where it is used, the `tag` column of `fundamentals.csv.gz` says so. **A stale value does not resolve.** An instant whose period end is more than 15 months before the snapshot date is rejected and the next tag in the list is tried. That is the same 15-month acceptability window the runbook applies to the annual TTM fallback, reused rather than a second figure being invented. Without it, a sparse cover-page tag hands back a share count years out of date and every ratio built on it is wrong. ### Coverage you should expect Measured on a sample of 25 names at three snapshots, 20 to 21 of 25 resolve completely across all seven base metrics. The names that do not resolve fail for reasons that are correct rather than fixable: - **Foreign private issuers reporting in a currency other than USD.** Ambev reports in Brazilian real, Abivax in euro. Both resolve to null rather than being converted. - **Depository institutions.** Banks frequently tag no top-line revenue concept at all, because "revenue" for a bank is interest income plus noninterest income and is not a single tagged figure. Their price-to-sales and net margin are null. - **Pre-revenue issuers.** A company with no sales has no revenue fact. Null is the honest answer, not zero. **Nothing is ever imputed.** Where no acceptable value resolves, the pack stores null. A null on a screen criterion is handled by the strategy's `insufficientDataRule`, not by a filled-in guess. ### Non-USD reporting Monetary metrics are accepted only in USD. An IFRS filer that reports solely in another currency resolves to null rather than being converted, because no currency conversion exists anywhere in this codebase and a euro-denominated revenue against a dollar market cap would produce a silently wrong P/S ratio. ### Coverage and screening eligibility **Screening eligibility follows fundamentals coverage.** A name whose XBRL resolves may be screened. A name that does not resolve stays fully tradable in list and market modes but is excluded from screens. The app states this plainly in its coverage display. No name is ever excluded from the price dataset because its fundamentals are missing. ETFs carry no XBRL fundamentals at all. They are fully tradable in every mode and are never eligible for a screen. This is structural, not a coverage gap. `coverage.csv.gz` reports `complete`, `partial`, or `none` per name per snapshot, over the seven base metrics, and the manifest records names resolved, names missing, and median filing lag per snapshot. --- ## 8. Sector labels Sector comes from the registrant's 4-digit SIC code, mapped to one of eleven labels through the committed file `datapack/universes/sic_sectors.csv`. That file is instructor-editable and version controlled, so a term's mapping is reproducible and the diff between terms is visible. A 2-digit group absent from the map resolves to `Unclassified` and is counted rather than silently dropped. The official SIC divisions are deliberately not used. Manufacturing alone would swallow roughly a third of a 1,000-name universe, and there is no Technology division at all, which makes a sector screen useless in practice. **A 2-digit map is coarse, and the compromises are visible.** These are the ones worth knowing before you write a sector filter: | SIC group | Mapped to | What lands there that you might not expect | |---|---|---| | 28 Chemicals and Allied Products | Health Care | Chemical majors sit with pharma and biotech, because pharma (2834) and biologicals (2836) dominate the group among filers | | 35 Industrial and Commercial Machinery and Computer Equipment | Industrials | Computer hardware makers (357x) sit with machinery | | 37 Transportation Equipment | Industrials | Automakers sit with aerospace | | 67 Holding and Other Investment Offices | Real Estate | REITs (6798) dominate the liquid names, so holding companies and blank-check shells land in Real Estate too | | 73 Business Services | Information Technology | Advertising (7311) sits with software and computer services | | 87 Engineering, Accounting, Research and Management Services | Industrials | Research-stage biotechs filing under 8731 sit with engineering firms | Sector is **filter-only**. It is categorical, so it has no direction to rank in and no z-score to weight, and the strategy schema rejects it as a ranking criterion. --- ## 9. Universe construction and its known gaps The equity universe is built each term by a deterministic rule: 1. Start from the SEC company ticker file, joined with the SEC's exchange file for the listing venue. 2. Keep common shares on NYSE, Nasdaq, and NYSE American. 3. Exclude ADRs where identifiable, units, warrants, rights, preferred shares, exchange-traded products and funds where identifiable, every ticker on the curated ETF list, and share classes beyond the most liquid class per issuer. 4. Rank by median daily dollar volume over the trailing 252 trading days. A name needs at least 126 observations in that window to be ranked (instructor decision, 2026-09-06), so a listing younger than six months, or a name the source holds only a stub for, waits for the next term's build. 5. Retain the top 1,000 and commit the dated CSV. **ADR identification is name-based and therefore incomplete.** SEC company data carries no ADR flag. The builder identifies depositary receipts from markers in the registrant name (ADR, ADS, American Depositary, Depositary Receipt or Share). Some ADRs will not carry those markers and will remain in the universe. Every exclusion is written to `universe_exclusions_.csv` with its reason so the rule can be audited rather than trusted. Ordinary US listings of foreign issuers are **not** excluded: that is deliberate, and their 40-F and 6-K filings are covered by the SEC XBRL API on the same footing as domestic filers. **Exchange-traded products are identified the same way, and for the same reason.** An ETF trust or a commodity fund registers with the SEC like an operating company, so the listing file offers it as a candidate. The first full build, on 2026-09-06, ranked SPY first and QQQ second in the equity universe and carried some twenty products in all, eight of them also on the curated ETF list. Two rules now keep them out. A ticker on `etf.csv` is excluded from the equity candidates outright, so the benchmark cannot be in the universe it is the benchmark for and no name can appear in both price files of one pack. Beyond that list, a product is identified from its registrant name (ETF, ETN, Fund, "Trust, Series", a metal or a coin followed by Trust, and the sponsors iShares, SPDR, ProShares, Grayscale, Bitwise, abrdn, VanEck, Direxion, Invesco QQQ, Sprott Physical and WisdomTree followed by Trust, ETF or Fund). "Trust" on its own is deliberately not a marker: REITs and trust banks carry it. A sponsor's own listed shares are an operating company, which is why Invesco, WisdomTree and Sprott are qualified by the product word. A product whose name carries none of those words stays in, exactly as an unmarked ADR does, and the exclusions file is where to look for it. **The name rules have false positives as well as gaps.** Preferred Bank (PFBC), an ordinary Nasdaq-listed bank, is excluded by the preferred-share rule because its name is the word. The rule was left as it is, since narrowing it needs a decision about which forms of the word mark a security, and the exclusions file makes the case visible each term. **Share-class deduplication keeps the most liquid class per issuer**, with ties broken by ticker ascending. So a dual-class issuer appears once. --- ## 10. Stated omissions in the simulation These are documented here and echoed in every results export. They all make a backtest look slightly better than reality: - No stock borrow fees on short positions, and no short rebate spread. - No market impact beyond the spread assumption. Your fills do not move the price. - No taxes. - No intraday path beyond daily OHLC. A stop is evaluated against the day's high and low, not against the sequence in which they occurred. - Margin is modeled at a captured broker schedule, not against live broker rules. The spread convention: `spreadBps` is the **full quoted bid-ask spread**, and crossing it costs **half of it per side**. At the defaults that is 2 bps commission plus 4 bps half spread, so 6 bps per execution and 12 bps round trip. --- ## 11. Pack integrity Every pack carries a `manifest.json` recording a SHA-256 for each file, row counts, ticker counts, the date range its price files cover, the truncation date, and `backtestableStart` and `backtestableEnd`: the window the pack is **for**, as opposed to every date it carries. Beside every zip the builder writes a `README.html`, a self-contained page rendered from that manifest, so the same facts can be read in a browser before the pack is loaded. A pack reaches back before that window so signal lookbacks are warm at the first rebalance, and the app offers the declared window rather than the calendar. The engine refuses to run against a pack whose file hashes do not match its manifest. A pack you build yourself from your own CSVs will load, and the app will say plainly that it is unverified: exports from such a run record `manifestSha256: null` and `official: false`, and the header carries an UNOFFICIAL PACK badge for the whole session. That path exists so you can keep using the tool after the course ends. It is not a path for coursework, and the audit runner flags any submission bound to it.