Some forecast models can be perfectly fitting the price for a period of time, but how long will they be correct?
Will the forecast model be accurate in a few months from now?
In this article I will show how I test my models over time. This is not a coverage of all tests but just a glimpse of the method.
I will use GOLD for this use-case.
I will use the price history up to Jan 2018 as learning period for the 5-Layer model. The rest of the price from Jan 2018 until today Feb 2019 will be used to verify the forecast by visually comparing the real price and the projection lines.
The first forecast will be from Jan 2018. I will then move the start of the forecast area to Sep 2018 and re-run the forecast models. Now i have two sets of projection lines which I can use to verify with the real price and also verify the consistency between them. I expect that the projection lines generated on Jan 2018 will be in line with those generated 9 months later, with minimum variance, and both will be in line with the real price.
Here is how the GOLD price looks like after loading:
In this use case I will use two layers from the 5-Layer model, the Yearly (1st-Layer) and the Weekly (5th-Layer).
Here is the result of the two layers projection lines from Jan 2018 (in dark blue):
I can now check visually if the real gold price is in line with the trend forecast indicated by the projection lines.
I am adding the real price from Jan 2018 without changing the projection lines. The 5-Layer model still assumes the price ends on Jan 2018.
As we can see, the real price trend is in line with the projection lines and shows a very good correlation. The general yearly trend and also the detailed weekly trends.
Now let's move the training period to Sept 2018 and run the 5-Layer model once more and generate the projection lines (in red).
We can check now the correlation between the real price and the projection lines:
I am now adding the real price without changing the 5-Layer model or the learning period.
Until now we have two sets of projection lines, one set (1st and 5th Layers) from Jan 2018 and one from Sep 2018. Each with a very good correlation to the real price.
But are they consistent with each other?
Let's add the two sets on the same view and see how they look:
The dark-blue lines are from Jan 2018, red lines are from Sep 2018.
As we can see: not only that each of the projection lines correlate with the real price as we have seen earlier, but they also in line with each other:
- The Yearly model (1st layer) from Jan 2018 is almost identical to the 1st layer from Sep 2018, with the exact dates of the change-of-directions.
- The Weekly model (5th layer) from Jan 2018 is almost identical to the 5th layer from Sep 2018, with the exact dates of the change-of-directions.
This use case shows that the 5-layer model on GOLD has very high correlation to the price and it is consistent over the tested period.