Beware of "good" systems - Walk Forward Analysis
Part II
Holy Grail
I would like to start this article with something that is quite unusual to me. I want to make some advertisement. It is about the software that is designed to create automated trading systems. Though all examples provided are done with Timing Solution software, it is not the advertising for Timing Solution. Actually, it will be a class regarding the verification of trading systems. Are you ready? Let's start.
Imagine, that we are in the year 2004. You do not know neither me, nor Timing Solution. You have just opened some advertisement. You see there:
INCREDIBLE POSSIBILITIES! UNIQUE FEATURES! DREAM OF ALL TRADERS FULFILLED! (There could be any words that may strike your imagination.)
Applying the latest math findings and the newest approach to trading, our Company has developed a new trading technology that is based on astro cycles (instead of "astrocycles", you may see here anything - math regular cycles, some indicator or some force of Nature, etc.) With this technology, you get a new trading system that catches precisely the smallest changes of the market price movement. Below is the screenshot of the equity curve for this trading system (the green line):
Within 16 years (since 1988 till now - now is 2004), this trading system has made 396 trades, and 67% of them are winning trades:
The well known parameter that is used to estimate the quality of automated trading system, System Quality Number (SQN), has a value of 9.956!!!! Remember that the systems with SQN higher that 7 are specified as Holy Grail systems, and we give you almost 10!!!
The average Annual return for this system is more than 30%. These are returns year by year:
(Then many other good things may follow, repeating by different ways the same numbers, with many exclamations; I will skip them.
Would you be interested in such a system? Interested enough to spend your time and money?
Holy Grail's analysis - Back Testing
That was the ad you have probably already seen somewhere. I tried to show some possible variations. As any ad, its purpose is to make you interested. But... You know, everything that was said in that advertisement is true! All calculations were correct, no mistakes there. Check it yourselves, you may download Timing Solution worksheet for that model here.
However, I would not recommend to apply this strategy for real trading. At least, some consideration is needed.
The first thing to try is to apply this model (this system, this strategy) to some other time span. As an example, you could read about that model nowadays, in 2011. Therefore, we can compare the projection line made by it to the real price movement, till 2011, and look at the results:
The equity curve is practically flat, and win loss ratio is 51%, i.e. the signals provided by that system are practically random signals. Annual profit is not good at all, though the annual return for the last year would be 41%. As we see now, this is definitely not a Holy Grail.
What we have done is practically a back testing analysis of that trading system. We performed it using a portion of the price history unknown to the system's developers. Looking back at these seven years, we can say to the developers: "Guys, we applied your Holy Grail system since 2005 and did not get so good results as you promised!".
If you would thinking about that system in 2005, I would recommend to try it on different time intervals, not just that one that is shown on their example. By the way, look at that example again. There the portion of price history prior 2004 is called "IN SAMPLE" interval and everything after 2004 is called "OUT OF SAMPLE" interval. So, looking at some new trading system, always ask about its performance on the out of sample interval. Spend some time to be sure that this is really out of sample interval, i.e. the system's vendor did not "improve" the system using some info from out of sample interval, just to get "better" results.
The main source of this error
I do not think that developers of Holy Grail system introduced here initially planned to cheat you. No, it is not that simple. The problem is that, while dealing with financial data, we face a very difficult for understanding phenomenon. We cannot rely on our common sense here, or, if we do, we should be prepared for very unexpected situations. Benoit Mandelbrot called this phenomenon "wild randomness", while our common sense is built on "mild randomness". You can find some info regarding this issue here: http://www.timingsolution.com/TS/Articles/cds/index.htm According to "mild randomness", a trading system that makes 266 winning trades versus 139 losing should work in the future without any doubts. If you add to your common sense classical statistics, it will not be very helpful: the classical statistics does not leave a smallest chance for the error in this case. In other words, if we apply normal statistical formula for these digits (266 win versus 139 loss), we will get practically 100% confirmation that this system is a true Holy Grail. However, the real financial data play a role of a dream breaker, telling us that this Holy Grail system is still another game of His Majesty Chaos. It means that impressive trading systems with impressive annual returns and great win/loss ratios may fail immediately after you apply them for the real data. All dreams disappear when we pass a magic point "Now".( Personally, it took me about two years to adjust my mind and accept this thing - after almost 20 years when I worked at the Institute of Nuclear Research and dealt mostly with normal, "mild", statistics).
Games of Chaos can be very tricky. I know a case when some trader X using technology Y made 9 winning trades (a real example)! People tend to think that there is some secret behind that. Their logic is pretty obvious: if somebody flips the coin 9 times, and all these 9 times he gets heads - that could not be an occasional fact, the probability of failing to get head 9 times in a row is 0.4%. This is what the classical, normal, "mild" statistics tells us. The most obvious explanation is that something is wrong with the coin, so we assume that in the future, with that same coin, we will get much more heads than tails. In the same manner we would expect more winning trades for the trader who made 9 winning trades in a row - and it does not matter how we explain it to ourselves, either by a unique personality of the trader, or by some magical system that he uses...
This way of thinking may be correct for the coin flipping (this is the base of "mild randomness"), while the stock market follows different rules (and this is "wild randomness"). Nine in a row winning trades is not a big event for financial data, though nine succeeding heads is a big event if we flip a coin. To illustrate this idea, I have created an astro based trading strategy that provides 55 winning trades versus 1 losing trade within the last four years; here it is:
Please do not take it as a trading advice, this is not a working strategy; if you try to trade with it, there is a very strong chance that you will loose your money. Here you can download Timing Solution worksheet for this model.
Walk Forward Analysis (WFA)
In the examples above we have considered two intervals - in sample and out of sample intervals. And I am telling you that this is not enough. There are a lot of cases when we may have good results on both these intervals, and it will be still the Game of Chaos, and not a system we can rely on. You may ask me a very legitimate question: Is it possible to verify an automated trading system somehow?
I see only one way - to model the process that is going in a real life: you purchased software, run it, get trading signals for your financial instrument, execute the trades and - the most important issue - observe the results on your account. This approach will definitely help you to figure out how reliable is your trading system, though it would be nice to find some other, less risky, method.
You can do this: from some moment, start collecting FUTURE signals generated by the trading system and perform paper trading based on these signals. Pay special attention - these signals should not be modified in any way, i.e. if for example yesterday this system has recommended to sell, and you made this sale, while today in the morning you have found that this system does not recommend to sell - you should ignore this info because sell signal is already executed -"there is no undo button in real life". You should follow this procedure for some time. Be sure that this system works on both - up and down - trends. I understand that this is a very time consuming procedure, though it models the real trading. In brief, this is a description of WFA - Walk Forward Analysis. Actually the programs should provide Walk Forward Analysis service. They do not do that because of two things (in my opinion):
1) this is very time consuming procedure;
2) the developers and users must be sure that this procedure does not contain any "future leaks" - which is in reality a very sophisticated task, as some portion of the new coming (future) price may be used to calculate some indicator, or non lag moving averages are used, etc.
In any case if somebody recommends you some trading system (including me), - you should ask how this system is verified. Ignore all high %% as a non important advertising information. In my experience, the small %% (like 55%-60%) of winning trades calculated correctly is much more valuable information than 80% annual return.