One more method of finding the best forecasting model

written by Sergey Tarassov

In this article I would like to share with you some method of finding the best projection line. You can consider this method as an alternative to Back Testing procedure. Back Testing usually takes a lot of time for calculations, while this alternative approach can be performed in less than one hour.

However, before any discussion, I prefer to clarify the subject of the forecast's accuracy. I have said it already and I will repeat it many times, because I am pretty much sure that the question of accuracy is the darkest question practically for any trading system available on the market. You can easily find in the Internet advertisements for the systems that promise 80% and even higher accurate forecast. At the same time you can find on many forums the complaints of the users for whom "these systems do not work". And they still look for other systems that promise again 80% and more...

I did the calculation how much money you would make if somehow you find the way to forecast next week price movement 80% accurate. Guess how much? You will be surprised: within just one year you can increase your capital to 1200% (E-mini analyzed)!!!!

I put the results of these calculations into the table. It shows your Annual return in regards to different forecast horizons and accuracy.

Now try to remember your own real Annual return and returns of your friends, for different financial instruments. Make conclusions yourself. 

Back to the theme of this article. The idea of this method is pretty simple: we organize the competition between the different forecasting models and pick up the best one. Remember that this method is applicable for financial instruments with long price history (at least 10 years). 

I will show how to do it for S&P 500 index. Download S&P 500 data from the year 1950; this is almost 60 years of price history. 

The first thing to do is to divide all available price history. The reason is to get two data sets: one - to train the Neural Net and another - to test the model:

I set LBC in this example somewhere in the middle. Thus we have 7288 price bars till the year 1980 year (it is our training interval) to optimize our model and 7106 price bars 1980-2007 y (the testing interval) to check how this model works. 

The trick is: you should make the testing interval as big as possible because the big testing interval allows to avoid occasional coincidences.

Next step is to create several forecasting models and then test them one by one.

Before creating any model, decide what you will forecast. Let us try to forecast the relative price oscillator, with the period of 100 bars:

The usage of oscillators  allows to eliminate any trend inside the data history.

Now we need to define the forecasting model, what it is based on. You can create all models during one session; set the other model using "+" button here:

Let's start with the astronomy based dynamic model:

I use all price history data (on the training interval) to optimize Neural Network. Click this button: 

Now click "Training" button and in 2-3 minutes after 10K-20K training steps you will get this:

As you know from the documentation and classes, the bottom diagram represents the relative price oscillator for S&P500 (a black line) and Neural Network projection line based on the chosen Dynamic model (a red line).

We need to find the model that forecasts best on TESTING interval, i.e. between 1980 and 2007 years. We do not use these bars to optimize Neural Network; therefore the coincidence/correlation between the price and projection lines shows us the real forecasting power of our models.

Click these buttons to calculate the correlation between the oscillator and the projection line on the TESTING interval.

The higher correlation the better coincidence between the oscillator and the projection line we have. I recommend to pay attention to those models that have a correlation higher than 0.1.

If you look at the projection line on the TRAINING interval, you will see a very good coincidence, but you should understand that this does not mean that the model provides a good forecast. The real forecast ability appears on the TESTING interval only.

Particularly this model does not provide the good forecast (as you see from the correlation coefficient). We will not work with it, so we clear it choosing "Delete All" item:

Then we create and check in the same way another model:

This model is based on a planetary speed (I do not use the Moon because our goal is the long term forecast).

After training, I have got this correlation:

This is not enough. 

I clear this model too and have downloaded another astronomy based model, waxing/waning model (I recommend to try its different variants):

After training I have got a better correlation for this model:

It is still a small correlation though it is better than its predecessors. In this case I recommend to look at this projection line in details. 

We can see here that this model provides pretty good forecast for the year 2007. 

I recommend also to try these models: 

To make a long story short, I have tried these models and got a not bad projection line with "Aspects Model"  with these options:

It gives this projection line:

Usually I analyze a dozen of models or so. However, do not waste your time trying to find the ideal projection line. 

I think that usually there are 2-3 good models for any financial instrument. I believe they reflect 2-3 possible scenarios of the stock market behavior.

Put all these projection lines into Strategy clicking this button:

and you can see all these projection lines together:

Also I recommend to draw Annual cycle together with these projection lines clicking this button: