Neural Network backtesting module

part of Terra Incognita project

Introduction 

Suppose we need to verify some Neural Network (NN) model for EOD data. The best way to do it is providing a backtesting (= walk forward analysis) for this model.

It works this way. Let's put yourself back 50-30 years back, i.e. imagine that you are in 1960s - 1980s instead of the year 2013. Let it be the beginning of the year 1965. And, being in 1965, we generate the projection line with NN module. When it is done (we have a projection line), we can watch (remember, we are in the year 1965?) how our forecast fits the real price. We are able to calculate the correlation between our projection line and the real price (to be exact, its oscillator) and use this parameter as a measure of "goodness/badness" of this forecast. The explanation regarding that correlation is here: http://www.timingsolution.com/TS/Mini/38/index.htm

Then we can imagine that we are in the year 1966 and repeat this routine. And do the same again for the year 1967, 1968, ... 2012 years. We can do that because we have all necessary information, and we can feed the system with controlled portions of data. In total, we can do this procedure 46 times and calculate the correlation for each year. Ideally, if we have a really good forecasting model, we may obtain 46 positive values of correlations. Consider it as 100%. From the other side, if we have 23 positive and 23 negative correlations it means that this model is not forecasting, it provides just a random fluctuation. And we consider it as 50%. We may also get 46 negative correlation values. It means that our model is also providing a good forecast, only inverted one. Give it 0%. So, whatever is our forecasting model, its correlation will be somewhere between 0 and 100%. Taking invertion into consideration, it will be between 50% and 100%.  

We (my small team and myself) have tested some models for daily data. AS so far, those models provide 55%-65% correlation. 

Let me show how you can test your models.

Scenario #1: backtest your model

1. You have to download 30 years of EOD price history at least. If you analyze Moon based models to identify the oscillations within a month, you need 4 years of price history at least.
   
2. Set Learning Border Cursor (LBC) in a way to have enough price history on TESTING interval (make Right mouse click on the Main screen and choose menu item "Set LBC ..."):
   
   
   
   
3. Create your NN model and click this button there:
   
   
   
4. Choose this template:
   
   
   
   
5. It takes some time for calculation. Please wait and when it is done, you should get the table like this one:
   
   
   
   
   This table provides the information about our imaginary experiment:
   We start to make a forecast on Dec 31, 1964, and this is a one year forecast - till December 29, 1965; the correlation -0.079 i.e. it is negative - the forecast is not good (we do not consider inversions here).
   Next year forecast (since Dec 29, 1965 till Dec 23, 1966) gives us a positive correlation  +0.076 i.e. the forecast is not bad. And then the next year, and the next one...
   
   In total we have 32 positive correlation versus 17 negative ones, i.e. 65%.
   

Scenario #2: find the best training interval

In example above we used 8 years of price history prior LBC to train our Network Work. The question is: how many price history do we need to get the best projection line - 3 years (750 bars),  4 years (1000 bars), 8 years (2000 bars) or something else? Using NN scientific language, the question is modified to this one: what training interval is the best for our model? For your information: most of EOD charts cover 250 price bars within one year.

In  order to answer this quiestion, you need to run this scenario:

You will get this table:

You see it shows that the best result 32 plus versus 17 minus gives NN with training interval: 2000 bars.

Just a reminder: you can define a training interval in NN module manually, clicking this button: