Beware of "good systems"
written by Sergey Tarassov
There was something that I wanted to discuss with you, users of my software, for years. I did not know how to approach this problem better. So, I have decided to start with this statement:
My friends, I am not responsible for the promises made by somebody else, not me. It is already 10 years as I receive several times a week the same e-mail. It may come from different people. The discussed item may be different. However, it is always the same story: somebody somewhere has invented some new forecast technique or trading method. That somebody promised some unbelievable performance of his/her system. The question to me is: what I think about it (which is OK) and can my/or Alphee's and my software do something similar to it. That last part of the question is what I want to discuss with you here.
Usually, these e-mails regarding somebody else's products have the link or
attachment. When I open it and try to figure
out what this system really does, almost always (with a very few exceptions) my
first thought is the one made by great Chinese philosopher Confucius more than 2 thousands years ago:"It is
difficult to find a dark cat in the dark room especially if there are no cats in
that room". And the same story happens again and again, every week, all
these 10 years.
.
In this article I will try to explain some moments you should pay attention to when you face any trading system/program/theory etc. (including mine as well). The first thing to remember is that there are no recognized formal criteria that allow to identify the workability of any product in this field. As to my knowledge, there are no government regulations here plus every creator of those systems and every provider of those services has a disclaimer on the first page. As so far, I see only one criterion to get some certainty about the workability of some market forecasting system. It sends us to Medieval Ages somewhere in the Middle East: "let the inventor trade with his system within a significant period of time; if this system does not work, cut his hand as a thief's" (of course, it is a joke).
However some recommendations can be made:
#1: Who are you? Before doing anything in regards to the new product/method etc., you should decide yourself what you are looking for. If you are ready to work systematically analyzing models and errors - you may try it and make your conclusions. If you are looking for some Miracle System, I guess you will have to spin the wheel of Fortune again and again, with all the consequences that follow, money included. This is another story.
#2: Post factum explanation Carefully read (or listen to) what is actually said about the product. Post factum explanations are the most impressive games of His Majesty Chaos. If somebody states something like: "with this system, we have found that five major turning points of the last year exactly coincided with %X astronomical/astrological/physical phenomena" - remember that in most cases it means that His Majesty Chaos has put that year's turning points and those phenomena into one box. This is an occasional fact, and nothing else. It does not mean that His Majesty Chaos will do the same the next year. We have to start to thinking in terms of statistics, otherwise we will have to play games with His Majesty Chaos again and again.
#3: Out of Sample Though there are no recognized formal criteria to evaluate the workability of the system, we have some means that can help us. Statistics provides us with them. "Out of sample" analysis is the first thing that should be done. For example somebody says that his model works very well till now (June, 2010). The first question you should ask is about the model's performance on "out of sample" interval, this is the most important parameter. Technically you leave some amount of data (the price history) not involved in the process of the analysis/optimization. These data should be used to check the performance of this model.
How to do that? Suppose today (June 1,2010) somebody is advertising some system that generates the projection line. You receive a very impressive pictures of how this system worked in the past. Simply skip this information as you need to be sure that "out of sample" analysis has been conducted correctly. Instead, ask that person to provide a forecast for the next several months. Put this forecast aside and wait. When the time passes (these months), compare the forecast with the reality. If the model provides buy/sell signals, ask to provide several future signals and record them. You should not blindly rely on the past performance information - first because of the fact the civilized society of the 21st century is more tolerable to the non verified information as the society of the medieval Middle East, plus in most cases these mistakes are non intentional.
If you do not like to wait, the correct analysis would be to apply the model for some other time interval. If it is the system, it should work! Try to use the price history since 1990 till the year 2006 for optimization, and the rest of of the available price history (2006-2010) can be used to check how that model works in reality.
#4: Sample Size. If somebody states that his/her system provides 80% accuracy forecast, with 8 win and 2 loss trades, - ignore this information as well. The minimal amount of trades to consider should 30 at least, better 100. Usually when advertising some techniques they provide several trades only; simply ask for the information about last 30 consequent trades. This is a minimum.
Even with this information, it is not a whole story. The most important part lies behind these digits. Financial mathematics has a lot of ugly nuances. The statistical criteria we use in our everyday life do not work when we apply them to real financial data. Benoit Mandelbrot called the statistics that works for financial data "wild randomness" in comparison to "mild randomness" we used to deal with (more info about this issue can be find here: http://www.timingsolution.com/TS/Articles/cds/index.htm). I highly recommend to contact with a professional in regards to this subject.
Look at this example. This is a statistical report for buy/sell strategy (S&P500 5 minutes chart, 2005-2009 years):
The system provides 60.9% win trades (283 win trades versus 182 loss); however this strategy does not provide a good profit:
The best profit is achieved by the strategy that provides 57% of win trades only.
Moreover we cannot consider a strategy with the highest profit as the best one. The profit can be done with several good trades when price has changed sharply while losing money all the time - still within that same system. Not many people can feel themselves comfortable with it.
There are tricky nuances of financial mathematics.
If we deal with the projection line, the best way is to calculate the correlation between the projection line and the price (do not forget to handle the trend, i.e. use detrended data). The interval to calculate the correlation should be:
a) out of sample interval;
b) big enough to avoid occasional errors (for daily data it should be at least 2 years, for intraday data at least a week).
#5: Multi trend. The system should be verified on different intervals with different trends. Better display the equity curve and the price chart together to see how the system overcomes periods of different trends.
One thing is to invent the system that made money in the years 2003-2007, and a totally different story is to find the system that still works within turbulent years 2007-2009.
#6: Use the formal criteria only. The system advertised should be available for a formal verification. The human factor should be avoided where it is possible. If somebody demonstrates the ability of his/her system performing trades where some subjectivity is involved, this is not a good sign. It complicates the whole thing. I have faced the situation when different traders made absolutely different decisions - looking at the same projection line. It is good to use your common sense, but there is a trick as well.
Let me explain what I mean. It is about the difference between formal and non-formal system. A formal system is clear - it has some set of operations that anybody can do under certain conditions. For such a system, it does not matter, who is the applicant. It should work, and it should provide the same result for you, for me, for other people - if we use the same initial data, do the same operations, follow the same rules.
A non-formal system is different. It may start as a formal system (like calculating the projection line). But there is a step in it where your next action is not defined. And you are suggested to use your common sense. And then you are surprised to see that you lose where your teacher definitely made a profit. If you have a possibility to see the teacher again, he fixes the problem in a second, and you say to yourself: "why I did not see it (or did not do it), it is so obvious". It is a case when the system that really works well for one person does not do so for the other.
What is going on here? When somebody explains how you can make a good profit using his/her trading system plus applying your common sense, we usually forget about the most important portion of this equation - "common sense". We do it simply because we all have it - but it is different for each of us! Applying our common sense to some system, we add to the game our brain. Yes, it is true that the human brain is the most powerful computer in the world. But do we actually know about it? Here are some interesting facts. The oldest part of the human brain, "reptilian complex", has started to form several hundreds millions years ago, it comes from ancient fish and reptiles. This part of brain is responsible for our orientation in the space and for basic survival skills. IMHO it allows to make the right decision skipping all logic details: while making typical trading action, we identify turning points (orientation in space), set protective orders (survival skills). The part that is responsible for the logics is started to form recently. This means that we should understand exactly what part of the brain we refer when make some trades based on some trading system. I think it is better to separate the knowledge based on our intuition and the knowledge received from formal systems such as software programs.
I know this for sure as the most difficult thing to program are basic operations that are obvious from the common sense's point of view (Fuzzy Logic mathematics is about that).
Dealing with any system, you should ask yourself whether you can repeat these trades having the same formal information (like projection line, indicators).
Afterward
This article has been written with one intention only: to save time when I get again a question regarding my opinion about some trading system. I will send a link to this article. There are no other intensions. I understand that suggested criteria are very strict. They are not my invention, I tried to follow the academic style used by Universities in regards to theories. If you really want to know something, you should concentrate on the most important subjects and put away all non important information. This is the only way to achieve something important in this life and not be drawn in details.
In the real life, you will be convinced to do the opposite, i.e. your attention will be attracted by many interesting and intrigued details like:
- very interesting ideas;
- some secret techniques;
- great bio of the inventor;
- simple ego/power games,
and many other issues that have only one purpose: to attract your attention. I highly recommend to put this info away and ask (at least yourself) the most important questions #1 ... #6.
Surely we can not describe the whole Universe with formal logics only, the "Magnum Ignitum" (Great Unknown) always presents. There is/it was/it will be something new that should be discovered, this is the main engine for all researches. But if we speak about some final results available for the public we should speak in terms of formal logics. It is your choice always - either the soap opera beauty of fantastic systems or victories on the real stock market with real problems (though it means to go through "... nothing ... but blood, toil, tears, and sweat. (W.Churchill, May 1940)".
June,1 2010
Toronto, Ontario