Back testing versus optimization

Introduction

The purpose of this article is to answer the most typical question I receive. Usually it goes this way: somebody somewhere invents some trading strategy and claims extremely good results; and every time when this novelty appears, one or another TS user asks me a question about it, especially how to evaluate such a thing. Before we go there, please remember  -  and this is very important - that I have nothing against of these models/techniques -as well as I have nothing for them. I am neutral. My experience says to me that some models can work while some can't. When I hear about some novelty, I simply do not know. My only intension is to show people some standard steps to verify these models, so they will be able to do that themselves and not ask me these questions.

When evaluating some technique/method/model that claims the ability to forecast the market, first thing that we must be sure of is this: it should apply non future leaks technologies. In other words, we should use the technologies that do not know the future but try to forecast it. Ensuring that there are no future leaks is a complicated business as future leaks have many faces. However, it is very important. If the technique has some future leak in its base, in reality it means that it may work very well for a demo example and stops working when you apply it to live trading. Here you will find an explanation how to verify trading models. We will discuss here two kinds of models: models that provide some projection line and models that provide buy/sell strategies, i.e. the strategies that produce ready buy and sell signals. 

Verification of projection lines

If somebody claims ability to make good forecasts, start with checking how many forecasts this person does on a regular basis. It is simple: just imagine that you are publishing several different projection lines every week; there always will be one that you can refer to after some time; you will always have a chance to say: "See, in March 2008 I made a forecast that gold will go up/down for that many points, and my forecast worked." This statement alone does not mean much. If you want to look at it seriously, the only useful information here is: some technique exists that might be able to forecast this market. "Might" - and nothing more than that. Only the reality can prove or disapprove this statement. I recommend to mark somewhere what this technique forecasts today, starting from this magic point NOW. It might be good to know the past, however we are more interested in the future. Save this this forecast and compare it to a reality in several months. I would recommend to do this procedure several times. I will not discuss this technique anymore because, after conducting such an experiment several times, you will be able to make your own comments on this issue. If it for daily data, you need to see forecast during several months. 

Another thing to remember is Annual cycle. I saw enough forecasts where the models were actually based on Annual cycle. This is how the annual cycle looks for DJI:

Very often these two simple things dictated by common sense are enough to make your conclusions - and without involving special knowledge and a very tiresome procedure of back testing.

Verification of buy/sell strategies - there is no "Undo" button in trading

Dealing with models that provide buy/sell strategies is another story (even if these are models that provide projection lines that are the base for trading strategies). I have to say that for years I could not find a simple technology suitable for verification of  buy/sell strategies. There is a standard approach that applies Walk Forward Analysis (WFA). But - this is actually a very time consuming procedure, and there are not many programs there that allow to perform WFA without future leaks. This is a very big issue, and I will not discuss it here. However, there are some things that may be helpful if you want to verify buy/sell strategies:

Advice #1 - always compare your buy/sell strategy with buy and hold strategy

While observing the equity curve, always remember the price chart. The return that your strategy provides is only a part of the useful information. For example, the strategy that has saved money in the hard year of 2008 when the stock market dropped so dramatically - that strategy may be much better than the strategy that makes money in the good year of 2009 (starting from March 2009).

I recommend to watch the equity curve (blue) and the price chart together (black): 

Watch how this equity curve works against the trend. As traders say. "a newcomer cares about the profit while a professional cares about the risk". 

Advice #2 - there is no "Undo" button in real trading

This is a very tricky question. It is easy to find some good strategy; trying to verify it is much more complicated. The most common mistake here is that usually people mix optimization and back testing. So, to handle this situation and avoid unexpected surprises from "Holy Grail" strategies, I recommend to use "there is no "Undo" button in real trading" approach. Here is the explanation:

- while downloading the price history, download not the whole price history, but its part only. For example, download the price history till August 2010. The price after August 2010 should be invisible for your program, this is not just extremely important, it is mandatory. If you have Timing Solution software, I can guarantee that  if you set Learning Border Cursor on August 2010, the program does not "know" what happens beyond this point. I am not so sure about other software. Better exclude a portion of price history after August 2010 manually;

- calculate last 2-3 trade signals according to the technology using the data that you have downloaded. Remember these signals, we will use them later;  

- now download more price history, let's say up to December, 2010. Using the same model, calculate buy/sell signals with new data and check those 2-3 previously saved signals. These signals should be exactly the sane (the same date and the same buy or sell status). If these signals are changing, it means that the method has "future leaks". This is similar to the situation when some trader has a counselor who recommends to buy some security and in a week calls back to this trader and says "Sorry, I wanted to tell you to buy now, not a week ago. The price has dropped a lot...". There is no "Undo" option here;

- repeat this procedure several times. 

I will demonstrate this approach using one example. Below you can see the projection line based on some dominant cycle (a red curve):

This model recommends very good trade: buy long on August 25, 2010 and sell in the beginning of November.

To verify this trade I have downloaded the price history till August 25, 2010 and re-calculated this projection line. This is what I have got:

Oops!!!! Where is my trade? The system shows that I should start it in the end of June, not in August. It means that to make a trade on August 25, 2010, I should know the price movement after that date. This is an indication that the model that I have applied has a future leak.

If you have Timing Solution software, you can easily handle this situation, and you will see the "true" projection line and trading strategy based on it that does not use any knowledge of the future movement.

Actually the described technology is a core idea of back testing technology, and it makes a huge difference between the optimization and back testing. The optimization procedure optimizes the profit, the results of the optimization sound like this: "If I would use some **** parameters for this trading strategy, I would get that much profit". Back testing procedure excludes any "would" from your vocabulary.

In reality projection lines or buy/sell strategies with future leaks look very impressive. Projection lines coincide with the price very well, buy/sell signals give a very good equity curve  - in the past. The problem appears when we apply these projection lines or buy/sell signals to real account; the result may be not so good.  But - in a week or so look at that same projection line or buy/sell strategy with new data added. You will be surprised to find that the projection line as well as buy/sell signals are corrected now and perfectly fit to newly added data. And the trade that would make you to lose your money only a week ago now has disappeared, and a new "winning" trade appears instead. For non future leaks technologies this situation is excluded, any executed trade will not disappear when new price data arrive.