047 - Tom Starke - Building a systematic process for development of systematic trading strategies
Why Research Must Precede Back-Testing
Grinold’s Fundamental Law of Active Management:
Information Ratio = Information Coefficient × √(Breadth)
Where:
Information Ratio (IR): This measures the risk-adjusted return of a portfolio manager’s active strategy, i.e., how much excess return (alpha) they generate relative to the risk they take.
Information Coefficient (IC): This is a measure of the manager’s skill or the correlation between their forecasts and the actual future returns. A higher IC indicates better forecasting ability.
Breadth (B): This refers to the number of independent investment decisions or opportunities a manager makes. In essence, it’s about how diversified the strategies are.
In short, for any amount of ‘skill’, alpha increases with diversification (of strategies, of bets, of markets)!
I know we all want “quick, actionable take-aways”, but the reality is that foundational principles of a strategy development process is at the core of successful trading, and you more than likely do not have half of this in place like you should. So, while this is ‘foundational’, and can only be covered briefly, don’t skimp on reviewing this stuff. It’s only in the Algo Collective that we’ll be able to take the time to deep-dive how to set this all up in a highly practical way. Believe me, once you have a pipeline for strategy development, you’re done! You churn out strategies that are more robust, quickly drop bad ideas and refine your portfolio quickly. You can focus on risk management, other research and constant review, while your trading takes place automatically in the background. At least, that’s my approach.
Part II with Dr Tom Stark here:
As I think about building a robust and efficient strategy development pipeline, probably the most critical and overlooked principles relate to the need to do research prior to back-testing. Too many traders rush into testing strategies without first validating their hypotheses or setting themselves on a sound quantitative platform.
A sound research process is where everything begins, and it’s what separates a disciplined quant trader from someone simply chasing the next flashy back-test. You need to build a hypothesis, test it, and then methodically attempt to invalidate it. Look at the process like a data scientist—not someone who just wants to brag about their best back-test result. Pause here. Have you honestly tried to invalidate before? Be honest with me! See how ego gets in the way? These things count my friend. They do.
The shortfalls of the ‘back-test to build’ methodology are substantial. Path dependency, missing trades, and hidden risks are just a few of the issues that arise when we treat back-testing as the starting point. I guess the main issue I have always seen with this approach is that it sort of assumes you ‘already have a strategy to test’ before you begin right?
How did you get said strategy?
The alternative: to test little ideas, piece by piece, allows you to ‘build from the ground up’. You’ll understand and trust your end-product so much more. A key issue with back-tests, as we mentioned in the show, are that they rely on a singular historical path, ignoring the trades we filtered out. Will these same sorts of trades be filtered in future? What if they aren’t? A disciplined process ensures that we test ideas from all angles, probing for weaknesses rather than simply running tests that confirm our assumptions. This approach helps mitigate the over-fitting that often plagues strategy development, giving you confidence in your system’s ability to perform in out-of-sample conditions.
Yep, unfortunately mindset and emotion still play a critical part in quant trading!
The key to success in systematic trading is having a repeatable, methodical process that allows you to build strategies that are both robust and adaptable. By focusing on research first, followed by rigorous testing and validation, you create a “production factory” for strategies. This factory doesn’t shy away from failed ideas but moves efficiently through them until you uncover strategies that work consistently.
So, does it help to bullet point a few ideas? Sure, but obviously not as much as taking this stuff, sitting with it, and making it your own, based on the skills and tools you currently have. The reality is that the process is different depending on the markets, time-frames and strategy-types you trade. Some guidance:
Our goal is to be grounded on a sound quantitative platform — we want to avoid fooling ourselves, but we also want to broaden our search capabilities to find fleeting edges. Begin with proper handling of data, using good software, understanding the markets. Understand how back-testing engines work.
Adopt an institutional mindset — I get this is hard to grasp initially but start by at least considering that there is a more institutional mindset. Focus on uncovering clues in data, not perfection; focus on risk; focus on your specific objectives; pay no heed to what others are doing; be ‘portfolio first’ minded; and essentially – build a development platform.
Set realistic expectations — build for where you are at and what you need. Spend some time building a set of strategies that play well together before even deploying a lot of capital live (deploy some for training). Run imperfect strategies (there’s no holy grail) and over time you’ll make them vastly more productive at a portfolio level.
Define objectives — clearly identify risk, diversification, markets, time-frames, and when you want the strategy to work along with when it’s expected to fail. What research premises is it based on? Document it.
Start with research before back-testing — do not jump directly into testing a ‘strategy’. Test behaviours in the market, unlinked to strategy concepts. See if narratives can be backed up statistically. Think about how you’d generate a strategy out of that knowledge. We live in a world where ‘truth is relative’, trading will show you that just aint so. ; )
Use hypothesis testing — create a hypothesis and conduct ‘proof of concept’ testing to validate the idea before moving on.
Test ideas from all angles — look for weaknesses and eliminate ideas that lack statistical significance. Look at all the trades, stress test ideas with ‘what if’ analysis that isn’t even in the back-test.
Avoid path dependency risks — ensure that the strategy works regardless of where the historical data starts or which trades are excluded from a back-test.
Understand and focus on risk management — “where could this go wrong?”. Don’t ignore that one test you did that showed everything behaving badly! This will drive you immediately toward ‘multi-model’ or ‘ensemble’ deployment where you try to be less ‘exactly right’ and more ‘right on average’. This is ‘institutional thinking’ at work.
Build a portfolio, not just a single strategy — you know this already. You aren’t building strategies in isolation, remember their role in the portfolio. If you’re just starting out, ignore this, build that one first thing.
Use robustness testing methods diligently — yes of course you need to deploy techniques such as Out of Sample, Walk Forward Analysis, Monte Carlo simulations, Variance Testing, and Stress Testing. These can be tools in the research phase not just ex post facto tests.
Keep strategies simple — complexity often leads to over-fitting. Aim for simplicity with well-executed strategies.
Keep costs down and be efficient — allocate capital efficiently, net out trades, reduce commissions. All this stuff becomes extremely important.
Build tools for the job — Dump data to Excel, or use Python or R or whatever it is for you, and streamline the analysis.
Understand the metrics that apply — know the performance metrics (e.g., Sharpe ratio, maximum draw-down (it’s just one number!) and how they relate to your strategy’s effectiveness. Different metrics for different objectives. Understand your tool kit.
Review and refine continually — iteratively improve the process based on research feedback and real-world performance data.
What is Research?
Research provides the blueprint for development. What inefficiency are you trying to exploit? Is there a plausible economic rationale for that? For instance, trend followers assume that price momentum persists due to under‑reaction; mean‑reversion strategies exploit temporary dislocations; statistical arbitrage looks for relative mispricings among correlated assets.
Conduct exploratory statistics: before back-testing, test whether the signal has predictive power. Use regressions, auto-correlation analyses or non‑parametric tests to see if returns respond to the signal across different assets and regimes. Evaluate parameter stability by plotting performance surfaces.
Research also embeds robustness testing into each component we test, so that, by the time we are robustness testing the strategy, we’ve already gained a lot of confidence in it. Say you have a hypothesis that extreme moves quickly revert: test every instance of it; over every possible ticker; in different regimes. Where do the results cluster? Over what kinds of stocks or futures? How correlated are they if they occur together? What time-frames are we operating in? What levels are important? Does the thing that works in period one also work in period two? What buckets can the results be divided into? Before you know it, you’re developing conclusions that result in meaningful strategy components & filters.
Research is the process of validating hypotheses, so it’s an effort in being scientific in our thinking. We want to identify assumptions, be open about them, don’t hide them subconsciously. The more we try to document the process we follow in our research, the more we can systematize and automate it. The faster we can work to validate ideas, the more we get done, and this is half the battle in such a competitive marketplace.
Here’s the key: if you have built the strategy on research, you can test it in context. Your view of it’s robustness out of sample is based on how many tests your ideas have already passed to get to this point. Basically, back-testing is just one of the tools in your tool-kit, it’s not the solution.
Conclusion: Research First, Back-Test Later
So why must research come before back-testing? Simple. Without a clear research foundation, your back-testing efforts are blind. Research:
Builds hypotheses that make sense in the market context.
Defines the risk-reward profile you’re chasing, so back-testing doesn’t mislead you.
Validates assumptions and prevents over-fitting.
Gives you confidence that your strategy is grounded in real-world data, not just market noise.
Put simply, back-testing a ‘finished model’ is a confirmation tool, not a discovery tool. Don’t rush into it. Prioritize your research, and only then will you have the clarity and insight to back-test with confidence.
Everything else I’ve got to say on the topic (and there’s a lot) will be built out in the Algo Collective community. Launching January 2026. I plan to deep-dive robustness testing methods inside the context of a systematic development path. See you on the inside.
Stay logical, listen to the outliers.
Simon
Get in Touch with Tom:
Dr Thomas Starke on Substack
« … look for weaknesses and eliminate ideas that lack statistical significance » D. Aronson demonstrated that triggers alone are not statistically significant. And that’s only when adding a filter that we can get some results able to overcome randomness. So here is my question: is an idea an event alone = a trigger, or does it include some filter already? As far as I understood, we should evaluate ideas on the baseline of all occurrences, that means unfiltered.