Neural Network backtesting module
part of Terra Incognita project
Suppose we need to verify some Neural Network (NN) model for EOD data. The best way to do it is providing a backtesting (= walk forward analysis) for this model.
It works this way. Let's put yourself back 50-30 years back, i.e. imagine that you are in 1960s - 1980s instead of the year 2013. Let it be the beginning of the year 1965. And, being in 1965, we generate the projection line with NN module. When it is done (we have a projection line), we can watch (remember, we are in the year 1965?) how our forecast fits the real price. We are able to calculate the correlation between our projection line and the real price (to be exact, its oscillator) and use this parameter as a measure of "goodness/badness" of this forecast. The explanation regarding that correlation is here: http://www.timingsolution.com/TS/Mini/38/index.htm
Then we can imagine that we are in the year 1966 and repeat this routine. And do the same again for the year 1967, 1968, ... 2012 years. We can do that because we have all necessary information, and we can feed the system with controlled portions of data. In total, we can do this procedure 46 times and calculate the correlation for each year. Ideally, if we have a really good forecasting model, we may obtain 46 positive values of correlations. Consider it as 100%. From the other side, if we have 23 positive and 23 negative correlations it means that this model is not forecasting, it provides just a random fluctuation. And we consider it as 50%. We may also get 46 negative correlation values. It means that our model is also providing a good forecast, only inverted one. Give it 0%. So, whatever is our forecasting model, its correlation will be somewhere between 0 and 100%. Taking invertion into consideration, it will be between 50% and 100%.
We (my small team and myself) have tested some models for daily data. AS so far, those models provide 55%-65% correlation.
Let me show how you can test your models.
Scenario #1: backtest your model
Scenario #2: find the best training interval
In example above we used 8 years of price history prior LBC to train our Network Work. The question is: how many price history do we need to get the best projection line - 3 years (750 bars), 4 years (1000 bars), 8 years (2000 bars) or something else? Using NN scientific language, the question is modified to this one: what training interval is the best for our model? For your information: most of EOD charts cover 250 price bars within one year.
In order to answer this quiestion, you need to run this scenario:
You will get this table:
You see it shows that the best result 32 plus versus 17 minus gives NN with training interval: 2000 bars.
Just a reminder: you can define a training interval in NN module manually, clicking this button: