The backtesting app: a beginner's guide

You are about to test an investing idea against real history. This guide assumes you have not invested before and explains every word as it comes up.

What you are actually doing

You are writing down a rule for picking shares, then asking the computer what would have happened if you had followed that rule in the past.

A rule might be "each month, buy the 20 companies whose share price has risen the most over the past six months". The app takes that rule, walks forward through history one day at a time, and works out what your money would have been worth at the end.

Backtest Running your rule against past prices to see how it would have done. It is not a prediction. It tells you how an idea behaved before, which is the only evidence you can get without waiting years.
The catch, and it is a big one A rule that did well in the past may do badly next year. Markets change. The value of this exercise is learning to build a rule carefully and to be sceptical about the result, not finding a rule that gets rich.

The words you need

Read these once. You will meet all of them in the app.

Share (also called stock or equity) A small piece of ownership in a company. If the company does well, people will pay more for that piece, so the price rises.
Ticker The short code a company trades under. Apple is AAPL. You will type tickers into the app when you want to name specific companies.
Portfolio All the shares you hold at one time, taken together.
Portfolio value (the app also calls this NAV) What everything you hold would be worth if you sold it today, plus any cash you have not invested. NAV stands for "net asset value", which is a formal way of saying "what it is all worth right now, after subtracting anything you owe".

The app starts your portfolio at a value of 1.00. If it ends at 1.20, your idea would have made 20 percent. Starting at 1 rather than at a dollar amount means everyone's results can be compared directly.
Return The percentage change in value. Up from 100 to 110 is a return of 10 percent.
Rebalancing Redoing your ranking and rebuilding the portfolio. If you rebalance monthly, then once a month the app re-sorts every company by your rule, sells what no longer qualifies, and buys what now does.
Going short (betting against a company) Borrowing shares you do not own, selling them, and hoping to buy them back cheaper later. You profit if the price falls. It is the opposite of normal buying, and the risk is larger: a share you own can only fall to zero, but a share you have bet against can rise without limit.
Backtest window (the app calls it "the dates you are testing over") The stretch of past time you try your idea on. Pick 2021 and the app pretends it is the first trading day of 2021, runs your rule forward day by day using only what was known at the time, and stops at the end of the year.

Your data pack decides what is available. It covers a fixed stretch, shown at the top of the app, and you may test over all of it or over any shorter stretch inside it.

A shorter window is not a smaller version of a longer one. Nine months of 2020 tells you how your idea did in a crash; three quiet years tell you something else entirely. Neither is the truth about your idea, which is exactly why your instructor will test it again on dates you have not seen. Those dates may sit earlier in the calendar than yours: a test is a stretch you did not tune on, not a stretch that comes afterwards. The next section shows this term's three windows on one timeline.
Benchmark The yardstick you are compared against. Here it is the S&P 500, an index of 500 large American companies, held through a fund with the ticker SPY. Beating it means your rule did better than simply buying the whole market.
Volatility How much the value jumps around day to day. High volatility means a bumpy ride. Two strategies can end at the same place with very different journeys, and the smoother one is generally the better one.
1.00 1.30 one year of trading days same ending value low volatility: higher Sharpe ratio high volatility: same return, lower Sharpe ratio
Both portfolios turn 1.00 into 1.30. The red one made you live through five big drops on the way. The Sharpe ratio is how the app rewards the green path for getting there calmly.
Drawdown How far you fell from your best point before recovering. A 30 percent drawdown means that at some stage your portfolio was worth 30 percent less than its previous high. This is the number that tells you whether you could have stuck with the strategy.
a high point 30 percent below it: the drawdown it recovers, but would you have waited? time
The drawdown is measured from the previous high, not from where you started. A strategy can finish well above its starting value and still have handed you a fall you would not have sat through.
Sharpe ratio Return earned for each unit of bumpiness. It answers "was the return worth the nerves?" Higher is better. Around 1 is respectable, above 2 is unusual and worth double-checking, and below 0 means you would have done better leaving the money in a savings account. This is the number your grade uses.
Bid-ask spread At any moment, buyers offer slightly less than sellers ask. The gap is the spread, and you pay about half of it every time you trade. It is small per trade and adds up fast if you trade often.
Margin Borrowing money from your broker to buy more than your cash allows. It multiplies gains and losses alike, and it costs interest. If your losses get too large, the broker sells your holdings without asking. That forced sale is a margin call.

Your data pack, and the term's three packs

Everything the app knows about the market comes from one zip file. Understanding what is in it, and what is not, is half of understanding your results.

Data pack A zip file your instructor posts. Inside are daily prices for about a thousand US companies and seventy funds, a snapshot of each company's accounts, and a few market-wide factors, all frozen on the day the pack was built. The app reads it on your computer and never sends it anywhere.
Read the page beside the zip first Next to every pack your instructor posts a web page, README.html, written by the same tool that built the pack. It says which dates you can backtest, how far back the prices go, how many companies are in it, which companies from the class list are not in it and why, and a fingerprint that the app shows in its header so you can tell packs apart. Five minutes with it saves an hour of wondering why a ticker is missing.

Two ranges, and why they differ

A pack carries more history than it lets you test. Suppose your rule ranks companies by their return over the past six months. On the first day of your window the app needs six months of prices that are already behind it, or the rule has nothing to work with. So the pack reaches back about two years before the window. Those earlier days are warm-up: the app uses them to get your rule ready, and it will not let you backtest over them.

warm-up history: read, never tested your window prices start here first date you may test last date
The page beside the zip and the header in the app both show these two ranges. The date boxes in the app only ever offer the blue part.

Why there are three packs, and why two of them come later

Your course grades your strategy over three stretches of history. You get the first one at the start of term: the design window. You build your rule against it, argue with it, and hand your design in. Only after that are the other two released: the test windows. Until then nobody in the class knows which dates they are, so nobody could have tuned a rule to them. That is the whole point of holding them back.

2022 2023 2024 2025 2026 Design Jan to Jun 2026 Test 1 Jan to Apr 2025 Test 2 Jul to Oct 2023 grey: warm-up carried by that pack
This term's three windows, each in its own pack with its own warm-up. The test windows sit earlier in the calendar than the design window.

Look at where the test windows fall. They are not after the design window; they are years before it. That surprises people the first time, because "test" sounds like "what comes next". It is allowed, and it is deliberate. A test is a stretch of history your rule was not tuned on, and a stretch from 2023 is exactly as unseen by your rule as a stretch from 2027 would be. It also has a practical advantage: the instructor can build every pack before the term starts and release each one on the day it is due, rather than waiting for the future to happen.

"But my design pack already has 2023 and 2025 prices in it" It does, as warm-up, because the design window needs two years of history in front of it. The app will not let you backtest those dates from the design pack: the date boxes stop at the design window. Could you unzip the file and look at them? Yes. Would it help you? Not really. You would not know which few months, out of two years, were going to be a test, and your instructor runs your frozen rule on the test packs themselves, not on anything you did. The honest use of the design pack is to design.
Vintage Two packs built on different days disagree slightly about the same dates, because the data provider restates past prices every time a company pays a dividend. So a result is only ever compared against the pack it was produced on, and the app writes the pack's fingerprint into every file you export. Do not take a number from one pack and expect a run on another pack to match it to the last decimal.
Survivorship The list of companies in every pack is the list that exists today, ranked by how heavily they trade today. Companies that failed years ago are not in it at all, and companies that only listed after a test window are not in that window's pack; the page beside the zip names them. Every strategy therefore looks a little better than real life would have been, and the older the window, the more so. Keep that in mind when a 2023 result looks strong.

Step by step, start to finish

The app asks you one question at a time. Answer it, press Next, and the next question opens; the menu on the left shows where you are. Your answers build up as plain English under the Strategy tab below the question.

Before you start: open the app and load your data Open the link your instructor posted, or double-click the file you downloaded. Then drag the data file (it ends in .zip) onto the big box at the top left. This takes a few seconds because it is reading years of daily prices. The data stays on your computer. Nothing is uploaded. Before you drag it, open the README.html posted beside it and read what the pack covers; the header of the app will then show the same dates and the same fingerprint, so you know you loaded the right one.
  1. Step 1: say who you are and what you believe

    Your name, your team, and a name for your strategy. Then the important part: the thesis.

    Thesis Your explanation of why your rule should work. Not what it does, but what you think is true about markets that makes it work.

    "I buy the 20 best performers each month" describes the rule. A thesis sounds like this: "When good news comes out, the price moves part of the way immediately and the rest over the following months, because large investors buy in stages. So recent winners keep winning for a while."

    This is graded An examiner can see whether you had a reason or picked settings until the number looked good. Write the thesis before you tune anything.
  2. Step 2: choose which companies you will pick from

    Three ways to do it. Pick one.

    MethodWhat it meansGood when
    Rank on company accounts Sort every company by things like profit or size, and keep the top slice. Your idea is about what kind of company to own.
    Name them yourself You type the tickers. Your idea is about specific companies you have researched.
    Everything Consider all companies, filtered only on price and how heavily they trade. Your idea is purely about price behaviour.

    There are also two optional boxes for names you want to hold no matter what the ranking says: one for companies you always buy, and one for companies you always bet against. Type tickers separated by commas.

    Finally, two filters worth understanding. Minimum price skips very cheap shares, which are expensive to trade. Minimum daily trading skips companies that barely trade, because in real life you could not buy or sell them without moving the price yourself.

  3. Step 3: choose how to rank them

    This is the heart of your strategy. You pick one or more measures, and the app sorts companies by them.

    For each measure you also choose a direction. "Prefer high" on momentum buys recent winners. "Prefer low" on volatility buys the calmest companies. If you use more than one measure, you set how much each counts.

    Start with one measure It is much easier to explain a result driven by one thing. Add a second only when you can say why it should help.
  4. Step 4: choose how many companies to hold

    Holding 5 companies means each one matters enormously. Holding 50 means no single company can hurt you much, but your best idea is diluted. Somewhere between 15 and 30 is a common compromise.

    You also choose how much of your money to invest. 100 percent means fully invested. Above 100 percent means borrowing, which the app will let you do and which will cost you interest.

    The app will stop you here if the numbers do not add up If you ask to hold 5 companies with no single one above 10 percent of your money, that only accounts for 50 percent. The app tells you exactly that, and offers you three ways to fix it. This is not a bug.
  5. Step 5: choose how much money goes into each

    The simplest choice is the same amount in each company. You can instead give more to the companies that scored best, or more to the calmest ones.

    The maximum size per company stops one holding taking over your portfolio.

  6. Step 6: automatic selling rules (you can skip this)

    A stop loss sells automatically if a holding falls a set percentage below what you paid. A trailing stop does the same but measures from the highest point since you bought, so it follows the price up. A take profit sells once you are up by a set amount.

    These feel safe and are not free: they sell during temporary dips and you pay to trade every time. Try your strategy with and without them and compare.

  7. Step 7: choose how often to trade

    Weekly, monthly, or every three months. Trading more often follows your ranking more closely and costs more.

    There is also a delay setting with a minimum of one day. This exists because in real life you cannot see today's closing price and also trade at it. The ranking is worked out using data up to and including the rebalance day, and the trades happen at the closing price of a later day. You cannot turn this off, and it is the single most important fairness rule in the tool.

  8. Step 8: choose the dates, and run it

    Above the form there are two date boxes: the day your test starts and the day it ends. They begin filled in with everything your data pack covers, and that is a perfectly good answer. Press Run and the results open on their own screen, with the record of your choices one tab away.

    You can also test a shorter stretch by moving either date inwards. You cannot go outside what the pack covers, and if you type a date that is outside it the app moves it back and tells you it did. The pack holds some extra history before the start date so that rules needing six months of past prices have six months to look at on day one; those extra days are not part of what you are testing.

    Shortening the window until the number improves is not research Every idea looks good over some stretch of the past. If you try 2021, then 2022, then the six months in the middle that worked, what you have found is the six months, not the idea. Choose your dates for a reason you could say out loud, run it, and write down what happened. Your instructor tests the same design on dates you did not choose, and that is the comparison that means something.

Reading your results

Read the red box first If there is a red box at the top of the results, read it before you look at any number. It tells you things like "your portfolio could not be filled" or "your lender forced you to sell". A great-looking return sitting above three warnings is not a great result.

The four big numbers

NumberWhat it tells youWhat to watch for
Sharpe ratio Return earned per unit of bumpiness. Your grade uses this. Above 2 usually means something is wrong, or the period was unusual.
Total return How much you made or lost over the whole period. Compare it with the market line on the chart, not with zero.
Volatility How bumpy the ride was. Above about 25 percent is a rough ride for a share portfolio.
Maximum drawdown The worst fall from a high point. Ask yourself whether you would truly have held on through it.

The charts

The first chart shows what one dollar would have become, with the market alongside for comparison. The second shows how far below your best point you were at each moment. The second chart is the one that tells you what holding this strategy would have felt like.

The event log

Everything the strategy did and why: every rebalance, every automatic sale, every forced sale. If something surprising happened, this is where you find out when.

Checking it in Excel

Press Download spreadsheet on the Print/Export tab and you get a normal Excel file with three tabs.

TabWhat is in it
Daily summaryOne row per trading day: portfolio value, cash, and how much was invested.
Daily positionsOne row per company per day: how many shares, the price they are valued at, what that comes to, and its share of the portfolio.
Trades and eventsEvery buy and sell, in plain English, with the reason.
Two checks worth doing

Row by row. On any row of the positions tab, shares times price should equal the value. Put =D2*E2 in a spare cell and compare it with column F.

Day by day. Filter the positions tab to a single date and add up the value column. Add the cash figure for that same date from the summary tab. The total should equal the portfolio value on that row.

If either check fails, tell your instructor: you have found a real bug.

Why the price does not match a price chart The price column is the adjusted price. It folds dividends back into the price, so a company paying you a dividend shows up as a gain rather than as a drop on the day it pays out. That is the price the app values your holdings at, which is why shares times price gives the value exactly. If you look the company up on a finance website the price there will be different, and neither one is wrong: they measure two different things.

A negative number in the value column means a position you have bet against. That is why the tab labels each row as either "Owned" or "Borrowed and sold".

Five ways to fool yourself

1. Trying settings until the number looks good

If you test fifty combinations and keep the best, you have found the one that happened to suit this stretch of history. Decide your rule from your thesis, then test it.

2. Ignoring trading costs

Turn the spread setting up from 8 to 30 and run again. If your result falls apart, your advantage was in getting a good price, not in the idea. This is the cheapest sanity check you have and it takes ten seconds.

3. Holding too few companies

With five holdings, one lucky company can carry your entire result. Look at the holdings table and ask what happens if you delete the best one.

4. Forgetting that the list of companies is a survivor list

The companies available to you are ones that still exist today, ranked by how heavily they trade today. Firms that collapsed years ago are not in the data at all, and firms that listed after a window are not in that window's pack. That makes every strategy look somewhat better than it would have been in real life, and more so on an older window: the 2023 test pack is missing about forty companies that did not exist yet, and the ones it has were chosen for being large now, not then.

5. Reading the return and skipping the drawdown

A strategy that ends up 40 percent after falling 60 percent along the way is not one most people could actually hold. The path matters as much as the destination.

What to hand in

  1. Your strategy file, ..._strategy.json, which holds your rules.
  2. Your results file, ..._results.json, which holds your numbers.
  3. The one-page printed summary, saved as a PDF from the print button.
Export both files from the same run Your instructor re-runs your rules on the same data and checks the answers match to nine decimal places. If you change a setting after exporting the results, they will not match and it will look like something was altered.

If you get stuck

What you seeWhat it means
The app will not let me continue past a step Something in that step does not add up. The message says which two settings conflict and how to fix it.
"does not match the checksum" Your data file is damaged or was edited. Download it again.
"This browser cannot save your work here" Nothing is being saved. Export your strategy file before you close the tab.
My portfolio is underfilled every time Your filters are leaving fewer companies than you asked to hold. Loosen a filter or hold fewer companies.
I lost my work Re-import the strategy file you exported. This is why you export early.

The full reference, with every formula written out, is in METRICS.md. Where the data comes from and what its limits are is in DATA_NOTES.md, which is assigned reading.