5 tips to reduce overfitting
Over-optimization (overfitting) is the biggest threat to AOS strategies and also a subject of endless discussions that have no simple, general solution. In today’s article, I would like to share with you 5 tips, how I myself try to minimize the danger of overfitting.
1. Build your systems using 40-50% of all historical data available to you. Leave the rest for robustness testing.
When I create my own strategies, I follow the rule to save as much data as possible. I try not to waste more data than is necessary to create a system, but I leave some of it for robustness testing. For this reason, to build a system I use only a part of all the historical data that is available. I usually have 10-12 years of historical data and for the building I use only 3-4 years. This period is different every time I build a new strategy (sometimes I use 2-5 years, sometimes 7-10). The reason is simple: the less data you use to create the strategy itself, the lower the risk of overfitting. I leave the rest of the data for my robustness tests. If the strategy seems viable in the small and limited sample of data, I start robustness tests using all the data. By using this method, I can be sure that the strategy is tested with the data that was not used for the construction of this strategy; this makes the test more objective and reduces the risk of overfitting. Of course, this approach gives me a lot of “garbage”, i.e. a lot of potentially interesting strategies that fail robustness tests. But this is part of developing high-quality ATS: just by being patient and disciplined, you get an edge over other traders, because most of them don’t have these qualities and tend to look for shortcuts that lead to finding a strategy. faster, however the quality is questionable. Finding a solid, high-quality strategy takes time and patience. And the less data we use to build the strategy, the lower the risk of overfitting.
2. A good strategy should work in other markets of the same category
The ideal strategy is the one that also works in other different and inconsistent markets. Finding such a “gem” is really extremely demanding and I am not far from the truth when I say that even if you search very intensively, it is a success to find 1 strategy per year that meets this criteria. Since we need to diversify our ATS portfolio, for practical reasons we cannot proceed at such a slow pace. And sometimes it is better to “loosen the rules a little” and accept strategies that pass all tests of robustness in other markets of the same category. Finding these strategies is much more realistic, and the frequency of finding a strategy like this reaches an acceptable rate. For example, I test a strategy created for the TF market so that it works also on ES, YM, EMD and preferably also on NQ, i.e. works on approximately all markets in an index category. Just like a strategy built for the TY market, I want it to do well in the US and FV markets, meaning to do well in all markets in the bond category. Before moving on to the next point, let me make an important note: sometimes it happens that during the verification of the strategy in other markets, you will find a situation where the strategy works even better in another market than the one for which it was created. In that case, you will be tempted to trade the strategy in this market. don’t do it. It’s just another form of optimization and these little optimizations add up and lead to big overfitting. Simply trade the strategy in the market it was created for, or trade all markets in the category. Learn to minimize attempts to choose the best of the range of acceptable results, as this already leads to overfitting.
3. Never modify a strategy after you have finished robustness tests and chosen live trading parameters
When the strategy passes all the robustness tests and chooses the correct parameters for live trading, do not modify the strategy any further! Parameters This is a high degree of overfitting after which the strategy will almost certainly fail. Yes, you will have good backtesting equity, but live trading results will be a huge disappointment. You should approach the entire optimization process with sensitivity and at all times make as few changes as possible. Changing parameters repeatedly to find the best ones leads to significant overfitting. Accept the fact that once you choose the parameters for your live trading, they are final and you should not change them. If you have more than one set of good performance parameters, you should prefer the average to the best. The best results tend to be overfitting. I would start trading live with all available parameter sets or choose the average.
4. Don’t skip the paper trading period
One of the simplest and most effective ways to eliminate a potentially overoptimized strategy is the paper trading period. When you complete the strategy, run it for 3 months on the mock account and see if the strategy follows your expectations (some of the traders I know go even for 6 months). You won’t gain any advantage by skipping this period or trying to find shortcuts. The only really invisible data (the actual out-of-sample data) are the ones that don’t exist yet. And that is exactly the reason why no final verification test can compete with the trading period on paper when we test the strategy on live data. Being impatient gets you nowhere. The best way to move forward is patience and discipline.
5. Don’t use too low time frames (to reduce “distortion”)
From my experience, I can say that the smaller the time frame you use to create a strategy, the more distortions there will be on the charts. How is it related to overfitting? Very simple. Good equity on a low time frame is more the result of overfitting than advantage. Higher time frames contain less distortion, therefore there is a better assumption that there is a real advantage behind the fairness we see. Personally, I use various time frames, but never shorter than 10 minutes. For day trading strategies, I usually use 15, 30, 45, but even 60 minute time frames. For my swing strategies, I go even higher: to 80, 120, 140, 180, 200 or even up to 240 minutes. I have multiple strategies over multiple timeframes, which I consider to be another form of diversification. A final note on time frames: check your final strategy on various time frames. You might find a time frame that gives you better results than the original, don’t trade it live, it’s another form of over-optimization. Use the time frame for which you have created the strategy and, in the worst case, choose the one with average results. All of the things I’ve described above seem like small things, but when you put them together, it’s a big step toward reducing overfitting.