You are about to test an investing idea against real history. This guide assumes you
have not invested before and explains every word as it comes up.
What you are actually doing
You are writing down a rule for picking shares, then asking the computer what would
have happened if you had followed that rule in the past.
A rule might be "each month, buy the 20 companies whose share price has risen the most
over the past six months". The app takes that rule, walks forward through history one
day at a time, and works out what your money would have been worth at the end.
Backtest
Running your rule against past prices to see how it would have done. It is not a
prediction. It tells you how an idea behaved before, which is the only evidence you
can get without waiting years.
The catch, and it is a big one
A rule that did well in the past may do badly next year. Markets change. The value of
this exercise is learning to build a rule carefully and to be sceptical about the
result, not finding a rule that gets rich.
The words you need
Read these once. You will meet all of them in the app.
Share (also called stock or equity)
A small piece of ownership in a company. If the company does well, people will pay
more for that piece, so the price rises.
Ticker
The short code a company trades under. Apple is AAPL. You will type tickers into the
app when you want to name specific companies.
Portfolio
All the shares you hold at one time, taken together.
ETF
An exchange-traded fund: one share that holds a whole basket of other shares, such as
SPY, which holds the 500 largest US companies. The header counts them separately from
single companies.
Basis point (bps)
One hundredth of a percent. Trading costs are small, so they are written in basis
points: 10 bps is 0.1 percent, or $10 on every $10,000 traded.
Portfolio value (the app also calls this NAV)
What everything you hold would be worth if you sold it today, plus any cash you have
not invested. NAV stands for "net asset value", which is a formal way of saying "what
it is all worth right now, after subtracting anything you owe".
The app starts your portfolio at $100. If it ends at $120, your idea
would have made 20 percent. Starting everyone at the same amount means results can be
compared directly.
Return
The percentage change in value. Up from 100 to 110 is a return of 10 percent.
Rebalancing
Redoing your ranking and rebuilding the portfolio. If you rebalance monthly, then once
a month the app re-sorts every company by your rule, sells what no longer qualifies,
and buys what now does.
Going short (betting against a company)
Borrowing shares you do not own, selling them, and hoping to buy them back cheaper
later. You profit if the price falls. It is the opposite of normal buying, and the
risk is larger: a share you own can only fall to zero, but a share you have bet
against can rise without limit.
Backtest window (the app calls it "the dates you are testing over")
The stretch of past time you try your idea on. Pick 2021 and the app pretends it is
the first trading day of 2021, runs your rule forward day by day using only what was
known at the time, and stops at the end of the year.
Your data pack decides what is available. It covers a fixed stretch, shown at the top
of the app, and you may test over all of it or over any shorter stretch inside it.
A shorter window is not a smaller version of a longer one. Nine months of 2020 tells
you how your idea did in a crash; three quiet years tell you something else entirely.
Neither is the truth about your idea, which is exactly why a design is tested again
on dates its author has not seen. Those dates may sit earlier in the calendar than
yours: a test is a stretch you did not tune on, not a stretch that comes afterwards.
The next section shows how that works.
Benchmark
The yardstick you are compared against. Here it is the S&P 500, an index of 500
large American companies, held through a fund with the ticker SPY. Beating it means
your rule did better than simply buying the whole market.
Volatility
How much the value jumps around day to day. High volatility means a bumpy ride. Two
strategies can end at the same place with very different journeys, and the smoother
one is generally the better one.
Both portfolios turn $100 into $130. The red one made you live through five big
drops on the way. The Sharpe ratio is how the app rewards the green path for getting
there calmly.
Drawdown
How far you fell from your best point before recovering. A 30 percent drawdown means
that at some stage your portfolio was worth 30 percent less than its previous high.
This is the number that tells you whether you could have stuck with the strategy.
The drawdown is measured from the previous high, not from where you started. A
strategy can finish well above its starting value and still have handed you a fall
you would not have sat through.
Sharpe ratio
Return earned for each unit of bumpiness. It answers "was the return worth the
nerves?" Higher is better. Around 1 is respectable, above 2 is unusual and worth
double-checking, and below 0 means you would have done better leaving the money in a
savings account. It is the number most people use to compare strategies, and the
one the fund page combines across windows.
Bid-ask spread
At any moment, buyers offer slightly less than sellers ask. The gap is the spread, and
you pay about half of it every time you trade. It is small per trade and adds up fast
if you trade often.
Margin
Borrowing money from your broker to buy more than your cash allows. It multiplies
gains and losses alike, and it costs interest. If your losses get too large, the
broker sells your holdings without asking. That forced sale is a margin call.
Your data pack, and why there can be more than one
Everything the app knows about the market comes from one zip file. Understanding what
is in it, and what is not, is half of understanding your results.
Data pack
A zip file you are given. Inside are daily prices for about a thousand US
companies and seventy funds, a snapshot of each company's accounts, and a few
market-wide factors, all frozen on the day the pack was built. The app reads it on your
computer and never sends it anywhere.
Read the page beside the zip first
Beside every pack comes a web page, README.html, written by the same
tool that built the pack. It says which dates you can backtest, how far back the
prices go, how many companies are in it, which companies from the full list are
not in it and why, and a fingerprint that the app shows in its header so
you can tell packs apart. Five minutes with it saves an hour of wondering why a ticker
is missing.
Two ranges, and why they differ
A pack carries more history than it lets you test. Suppose your rule ranks companies
by their return over the past six months. On the first day of your window the app
needs six months of prices that are already behind it, or the rule has nothing to work
with. So the pack reaches back about two years before the window. Those earlier days
are warm-up: the app uses them to get your rule ready, and it will not
let you backtest over them.
The page beside the zip and the header in the app both show these two ranges. The
date boxes in the app only ever offer the blue part.
Why there can be more than one pack, and why some come later
A strategy is worth more when it holds up on history its author never looked at. So a
backtest is often run in two parts. First you get a pack for the design
window: you build your rule against it, argue with it, and freeze your
design. Only after that are further packs released, one per test
window. Until then nobody knows which dates they are, so nobody could have
tuned a rule to them. That is the whole point of holding them back. How many test
windows there are, and where they fall, is decided by whoever runs the exercise, not
by the app.
A design window and a test window, each in its own pack with its own warm-up. A test
window can sit anywhere in the calendar the design window does not, earlier or later.
A test window need not come after the design window; it can sit years before it.
That surprises people the first time, because "test" sounds like "what comes next".
It is allowed, and it is deliberate. A test is a stretch of history your rule was not
tuned on, and a stretch from years earlier is exactly as unseen by your rule as a
stretch from next year. It also has a practical advantage: every pack can be built in
advance and released when it is wanted, rather than waiting for the future to happen.
"But my design pack already has earlier years in it"
It does, as warm-up, because the design window needs history in front of it. The app
will not let you backtest those dates from the design pack: the date boxes stop at
the design window. Could you unzip the file and look at them? Yes. Would it help you?
Not really. You would not know which few months, out of years of warm-up, were going
to be a test, and a frozen rule is run on the test packs themselves, not on anything
you did. The honest use of the design pack is to design.
Vintage
Two packs built on different days disagree slightly about the same dates, because the
data provider restates past prices every time a company pays a dividend. So a result
is only ever compared against the pack it was produced on, and the app writes the
pack's fingerprint into every file you export. Do not take a number from one pack and
expect a run on another pack to match it to the last decimal.
Survivorship
The list of companies in every pack is the list that exists today, ranked by how
heavily they trade today. Companies that failed years ago are not in it at all, and
companies that only listed after a test window are not in that window's pack; the
page beside the zip names them. Every strategy therefore looks a little better than
real life would have been, and the older the window, the more so. Keep that in mind
when a result from years ago looks strong.
Step by step, start to finish
The app asks you one question at a time. Answer it, press Next,
and the next question opens; the menu on the left shows where you are. Your answers
build up as plain English under the Strategy tab below the
question.
Before you start: open the app and load your data
Open the app's address, or double-click the file you downloaded. Then
drag the data file (it ends in .zip) onto the big box at the top left.
This takes a few seconds because it is reading years of daily prices. The data stays
on your computer. Nothing is uploaded. Before you drag it, open the
README.html beside it and read what the pack covers; the header of
the app will then show the same dates and the same fingerprint, so you know you
loaded the right one.
Step 1: say who you are and what you believe
Your name, your team, and a name for your strategy. Then the important part: the
thesis.
Thesis
Your explanation of why your rule should work. Not what it does, but what
you think is true about markets that makes it work.
"I buy the 20 best performers each month" describes the rule. A thesis sounds like
this: "When good news comes out, the price moves part of the way immediately and
the rest over the following months, because large investors buy in stages. So
recent winners keep winning for a while."
This is what a reader judges
Anyone reading your design can see whether you had a reason or picked settings
until the number looked good. Write the thesis before you tune anything.
Step 2: choose which companies you will pick from
Three ways to do it. Pick one.
Method
What it means
Good when
Screen: rank on fundamentals
Sort every company by things like profit or size, and keep the top slice.
Your idea is about what kind of company to own.
List: name the tickers yourself
You type the tickers.
Your idea is about specific companies you have researched.
Market: everything
Consider all companies, filtered only on price and how heavily they trade.
Your idea is purely about price behaviour.
There are also two optional boxes for names you want to hold no matter what the
ranking says: one for companies you always buy, and one for companies you always
bet against. Type tickers separated by commas.
Finally, two filters worth understanding. Minimum price skips very
cheap shares, which are expensive to trade. Minimum average dollar
volume skips companies that barely trade, because in real life you could
not buy or sell them without moving the price yourself. Leave either empty to turn
it off.
Step 3: choose how to rank them
This is the heart of your strategy. You pick one or more measures, and the app
sorts companies by them.
For each measure you also choose a direction. "High is better" on momentum buys
recent winners. "Low is better" on volatility buys the calmest companies. If you use
more than one measure, you set how much each counts. The record on the Strategy
tab says in words which companies each choice prefers.
Composite
With more than one measure, the single combined score each company gets, which is
what the ranking actually uses. Each measure counts in proportion to its weight.
Z-score
How far a company sits above or below the average on one measure, counted in
standard steps rather than in the measure's own units, so measures of different
sizes can be added. "Weighted z-score" combines the measures this way; "weighted
rank" combines their places in the queue instead.
Start with one measure
It is much easier to explain a result driven by one thing. Add a second only when
you can say why it should help.
Step 4: choose how many companies to hold
Holding 5 companies means each one matters enormously. Holding 50 means no single
company can hurt you much, but your best idea is diluted. Somewhere between 15 and
30 is a common compromise.
You also choose how much of your money to invest. 100 percent means fully
invested. Above 100 percent means borrowing, which the app will let you do and
which will cost you interest.
The app will stop you here if the numbers do not add up
If you ask to hold 5 companies with no single one above 10 percent of your money,
that only accounts for 50 percent. The app tells you exactly that, and offers you
three ways to fix it. This is not a bug.
Step 5: choose how much money goes into each
The simplest choice is the same amount in each company. You can instead give more
to the companies that scored best, or more to the calmest ones.
The maximum size per company stops one holding taking over your portfolio.
Step 6: automatic selling rules (you can skip this)
A stop loss sells automatically if a holding falls a set
percentage below what you paid. A trailing stop does the same but
measures from the highest point since you bought, so it follows the price up. A
take profit sells once you are up by a set amount.
These feel safe and are not free: they sell during temporary dips and you pay to
trade every time. Try your strategy with and without them and compare.
Step 7: choose how often to trade
Weekly, monthly, or every three months. Trading more often follows your ranking
more closely and costs more.
There is also a delay setting with a minimum of one day. This
exists because in real life you cannot see today's closing price and also trade at
it. The ranking is worked out using data up to and including the rebalance day, and the trades happen at the closing price of a later day. You cannot turn this off, and it is the single most
important fairness rule in the tool.
Step 8: choose the dates, and run it
Above the form there are two date boxes: the day your test starts and the day it
ends. They begin filled in with everything your data pack covers, and that is a
perfectly good answer. Press Run and the results open on their own screen, with
the record of your choices one tab away.
You can also test a shorter stretch by moving either date inwards. You cannot go
outside what the pack covers, and if you type a date that is outside it the app
moves it back and tells you it did. The pack holds some extra history before the
start date so that rules needing six months of past prices have six months to
look at on day one; those extra days are not part of what you are testing.
Shortening the window until the number improves is not research
Every idea looks good over some stretch of the past. If you try 2021, then 2022,
then the six months in the middle that worked, what you have found is the six
months, not the idea. Choose your dates for a reason you could say out loud, run
it, and write down what happened. The same design can then be run on dates you
did not choose, and that is the comparison that means something.
Reading your results
Read the red box first
If there is a red box at the top of the results, read it before you look at any
number. It tells you things like "your portfolio could not be filled" or "your lender
forced you to sell". A great-looking return sitting above three warnings is not a
great result.
The four big numbers
Number
What it tells you
What to watch for
Sharpe ratio
Return earned per unit of bumpiness. The one number to compare strategies on.
Above 2 usually means something is wrong, or the period was unusual.
Cumulative return
How much you made or lost over the whole period.
Compare it with the market line on the chart, not with zero.
Volatility
How bumpy the ride was.
Above about 25 percent is a rough ride for a share portfolio.
Maximum drawdown
The worst fall from a high point.
Ask yourself whether you would truly have held on through it.
The charts
The first chart shows what $100 would have become, with the market alongside for
comparison. The second shows how far below your best point you were at each moment.
The second chart is the one that tells you what holding this strategy would have felt
like.
The event log
Everything out of the ordinary, and why: every automatic sale (a stop, a trailing
stop, a take profit, a price level you set), every forced sale, every margin call, and
every company that stopped trading. If something surprising happened, this is where
you find out when. If nothing did, it says so.
The ordinary rebalances are not in it. They are listed under
Diagnostics, just below: each rebalance day, how much of the portfolio
changed on it, and how many companies were ranked or left out.
The other words on the results screen
The results use the standard names for each figure, so you will meet them again
anywhere investing is discussed. Each one, in a sentence:
Sortino ratio
Like the Sharpe ratio, but it counts only the falls as bumpiness, not the rises.
Calmar ratio
The yearly return divided by the worst fall from a high point. How much you earned for
the worst stretch you had to sit through.
Beta
How much your portfolio moved when the market (SPY) moved. A beta of 1 moves with the
market; 0.5 moves half as much; above 1 moves more.
Alpha
The return left over after taking out what the market's moves explain. In the metrics
it is a yearly percent; in the factor tables it is a daily fraction, so the two
numbers look very different while saying the same kind of thing.
Tracking error
How far your daily returns strayed from the market's, over a year. Low means you
mostly followed the market.
Information ratio
How much you beat the market by, divided by the tracking error. Beating the market
steadily scores higher than beating it in lurches.
Turnover
How much of the portfolio was replaced, over a year. A turnover of 1 means the whole
portfolio changed once. Each change costs trading money.
Hit rate
The share of days on which the portfolio went up.
Fama-French three factor and Carhart four factor
Two standard ways of asking what your returns resemble. Each row says how much your
daily return moved with one factor: mktRf the market over the safe
rate, smb small companies against large ones, hml
cheap companies against expensive ones, and, in the four-factor version,
umd recent winners against recent losers. A momentum strategy shows
up as a large umd.
R squared
How much of your day-to-day movement the factors explain, from 0 (none) to 1 (all of
it).
Std error
In the factor tables: how far the estimate beside it could be off, given how noisy
the daily returns were. An estimate of 1.0 with a std error of 0.5 could easily be 0.5
or 1.5.
t
The estimate divided by its std error. Far from zero, beyond about 2 either way, it
means the estimate is unlikely to be chance in this window. It is not proof that the
same will hold in another window, or with money you have not yet invested.
Newey-West
The method behind the "std error" column. It allows for returns that run in streaks,
which would otherwise make the factor estimates look more certain than they are.
Left out, a measure missing
In the diagnostics: companies that could have been ranked but were missing one of your
measures on that date, so they were not ranked at all.
Checking it in Excel
Press Download spreadsheet on the Print/Export tab and you
get a normal Excel file with three tabs.
Tab
What is in it
Daily summary
One row per trading day: portfolio value, cash, and how much was invested.
Daily positions
One row per company per day: how many shares, the price they are valued at, what that comes to, and its share of the portfolio.
Trades and events
Every rebalance and every automatic sale, in plain English, with the reason.
Two checks worth doing
Row by row. On any row of the positions tab, shares times price
should equal the value. The first rows are a note and the headings, and the data
starts on row 4: put =D4*E4 in a spare cell and compare it with
F4.
Day by day. Filter the positions tab to a single date and add up
the value column. Add the cash figure for that same date from the summary tab. The
total should equal the portfolio value on that row.
If either check fails, you have found a real bug: report it to whoever gave you the app.
Why the price does not match a price chart
The price column is the adjusted price. It folds dividends back into
the price, so a company paying you a dividend shows up as a gain rather than as a drop
on the day it pays out. That is the price the app values your holdings at, which is why
shares times price gives the value exactly. If you look the company up on a finance
website the price there will be different, and neither one is wrong: they measure two
different things.
A negative number in the value column means a position you have bet against. That is
why the tab labels each row as either "Owned" or "Borrowed and sold".
Five ways to fool yourself
1. Trying settings until the number looks good
If you test fifty combinations and keep the best, you have found the one that happened
to suit this stretch of history. Decide your rule from your thesis, then test it.
2. Ignoring trading costs
Turn the spread setting up from 8 to 30 and run again. If your result falls apart, your
advantage was in getting a good price, not in the idea. This is the cheapest sanity
check you have and it takes ten seconds.
3. Holding too few companies
With five holdings, one lucky company can carry your entire result. Look at the
holdings table and ask what happens if you delete the best one.
4. Forgetting that the list of companies is a survivor list
The companies available to you are ones that still exist today, ranked by how heavily
they trade today. Firms that collapsed years ago are not in the data at all, and firms
that listed after a window are not in that window's pack. That makes every strategy
look somewhat better than it would have been in real life, and more so on an older
window: a pack for a window several years back can be missing dozens of companies
that did not exist yet, and the ones it has were chosen for being large now, not
then. The page beside each pack lists what it does not carry.
5. Reading the return and skipping the drawdown
A strategy that ends up 40 percent after falling 60 percent along the way is not one
most people could actually hold. The path matters as much as the destination.
What to keep, and what to pass on
Your strategy file, ..._strategy.json, which holds your rules.
Your results file, ..._results.json, which holds your numbers.
The one-page printed summary, saved as a PDF from the print button.
Export both files from the same run
Anyone with your strategy file and the same pack can re-run your rules and check that
the answers match yours to nine decimal places. If you change a setting after
exporting the results, they will not match and it will look like something was
altered. Export all three files does this for you: it saves
your strategy, its results and its daily returns from the run on screen, named for your
team and you. After a change it waits until you run the test again.
If you get stuck
What you see
What it means
The app will not let me continue past a step
Something in that step does not add up. The message says which two settings conflict and how to fix it.
"does not match the checksum"
Your data file is damaged or was edited. Download it again.
"This browser cannot save your work here"
Nothing is being saved. Export your strategy file before you close the tab.
My portfolio is underfilled every time
Your filters are leaving fewer companies than you asked to hold. Loosen a filter or hold fewer companies.
I lost my work
Re-import the strategy file you exported. This is why you export early.
Taking it further in Python
Once you know how your strategy behaves, the Print/Export tab can give you the same
strategy as a small Python project: a zip holding your design written out in Python,
the app's own backtest rewritten in plain Python, and a guide called
NEXT_STEPS.md that takes you through it.
It is for when you want to go past what the app lets you choose: add a signal of your
own, try a different set of companies, or trade the strategy. Its first command runs
the app's test of your design again on your own computer and checks it, day by day,
against the results file you exported, so you can see the translation is exact before
you change anything.
What you need: Python 3.10 or later, from python.org. To trade, you also need an
account at Interactive Brokers with paper trading, where the money is not real, and
their TWS or IB Gateway program running on your computer. Prices for trading come from
a free source whose limits the project explains. Start with the paper account.
The project can also trade a live account, with real money, if you choose to; that is
entirely at your own risk.
A second project, for paid data, sits beside it. It is the same project with prices
from a data service you pay for, or from Interactive Brokers itself, and it can trade
through either of Interactive Brokers' two ways in for programs. Choose it when you
are ready to pay for data before trading real money.
A third project is for QuantConnect, a hosted platform you pay for by the month, where
you edit, backtest and trade from your browser with nothing to install. Its notebook
checks the translation on your data pack there, exactly as the first project does. Its
trading algorithm uses the platform's own prices and list of companies, so its backtest
comes close to the app's without matching it, and the project marks every place it
differs.