Let's try to put it all together. Hopefully after reading this lesson you will have a good idea what to look for in algo-trading.
There is no holy-grail in auto-trading. With all tools we use here, we can easily generate a strategy with a nice steep equity curve, that goes 'to-the-moon', but this will be pure curve-fitting. Instead we want to concentrate on the following two most important things: statistical significance and system stability (robustness), and not the highest profits (as many of you would think). The main goal is to find as many as possible, stable and robust strategies that make some profits over time and add them to our portfolio of strategies.
Let's go back to our theory..let's create a perfect strategy with a WinRate of 80% (thick golden line) and compare it 500 randomly generated strategies (the figure below shows only first 100 random curves). Let's also print a histogram of all trades to see where the result of our strategy is placed in the probability distribution curve.
Figure 4.1: Results distribution of 100 trades
We can see that after 100 trades, our 80% WinRate strategy (golden line) if far from the 'random noise' dominated by the results of 500 random systems. The resulting blue bin of our golden line which is visible in our histogram is very small since the occurrence is only 1 (we have only one outcome of our winning system and 500 outcomes of randomly generated systems).
At this moment we see that our golden line result is statistical significant (since it beats all 500 random systems after 100 trades), but we still cannot tell if our result is curve-fitted or not. The only possible way to find it out is to perform several robustness tests on our golden line strategy. Here for we can do the following, e.g.:
During our validation period we will 'jiggle' (= randomize) many different parameters around our strategy and strategy settings to check for robustness. This method is know as Monte-Carlo analysis method which is widely used in science. We will use that a lot since it is one of the most important stability tests.
The perfect strategy:
Let's imagine that our golden line strategy is perfect and none our Monte-Carlo tests has any influence on the final outcome. In that case the final result will be always the same, so for example after 50 different randomized sweeps, our probability histogram will look as follows:
Figure 4.2: Perfect strategy distribution
Now we can see, that our blue bin is much larger since all of 50 randomized sweeps result in the same outcome. This means that our strategy is not affected by any of 50 random tests we could think of. In scientific terms we could say that our Signal-to-Noise ratio is very high (Signal=our golden line results, Noise=random strategies). Our strategy is perfect!
However in real life, this will be very different, since no strategy is perfect and we will run much more randomized sweeps in our Monte-Carlo analysis. (StrategyQuant has several different robustness test modes which we will use extensively in our strategy validation). This all will result in a 'cloud' of golden lines, where each line represent outcome of different randomized sweep. Our probability histogram will also look very different, see the following figure:
Figure 4.3: Real strategy distribution
Up to now we have seen that:
Those three golden rules can be put into one beautiful equation that describes the resulting total statistical significance:
Statistical significance ∝ ( Number of trades * degrees-of-freedom ) / Robustness test results variance
In simple words this means that the total statistical significance (higher value is better) is proportional to the total number of trades (more is better) within your testing period and also proportional to total amount of degrees-of-freedom (input parameters = less is better), and inverse-proportional to the observed variation of the outcomes of Monte-Carlo tests (less variance is better).
HOWEVER: Statistical significance is only one critical part in algo-trading. There are much more characteristics a profitable system should have. In practice all those characteristics are validated using well designed strategy validation tests. So, to make this conclusion complete, a good strategy should pass at least the following additional validation tests:
As we can see this means, that in order to validate 1 good strategy a huge amount of computation work is required. And for most of the people this level of programming skills is 'out of reach'. That is why we need good algo-trading tools. Tools that automate every single step required to generate, validate and identify all good strategies. On this forum everything we do is based on this one algorithmic trading tool: see next lesson.
Now we know what are the key elements we should be looking for during the strategy design period. You see that it is not easy, but totally possible..and we will tell you how: Read next lesson.
Why do the MC's have to be done separately? Can it not be done in one MC test?
Yes it can all be done in one step, however I am also interested in strategies that did not pass each single MC step, I always try to understand why... so doing this in separate steps adds a little bit of additional diagnostics of the entire workflow.
So I do it because of informative reasons not functional. That’s how I learned it, but of course you should do what works the best for you.
Moreover the results of some tests like randomization of data (random trades) contain very usefull info like the maximum expected DD level which you can use as a threshold to kill or not to kill your strategies in real trading.
Thank you for explaining. That does make sense as it can help to understand where it failed instead of just that it did fail.
I was doing MC last night. One thing I want to make sure of. With the number outputs (WF Net Profit, Drawdown, etc.), I understand that. You can use these numbers and compare to the backtest to determine its robustness.
But on the graph, what am I looking for? Am I looking for all the lines to be relatively the same as the main backtest's line? In other words, they are grouped together with very few or little outliers? So for example, the attached is an example of a bad one because the lines are not grouped tightly, right?
Yes basically, I look for systems that have very 'tight outcome spread' in Y and X direction. Ideally all outcomes should be grouped together and also be symmetrical vs main equity line. Your picture shows a very good example of a very bad system.
Additionally I look for a reasonable RetDD number (>> 0) and reasonable DD, at the following confidence levels: 95%, 95% and 100%. Below an example of a good system.