Are you sure?

In my opinion Degrees of freedom decrease Statistical Significance .. not enhance

**Statistical significance ∝ ( Number of trades * degrees-of-freedom ) / Robustness test results variance**

In simple words this means that the **total statistical significance** (higher value is better) is proportional to the **total number of trades** (more is better) within your testing period and also proportional to **total amount of degrees-of-freedom** (input parameters = less is better), and inverse-proportional to the observed **variation of the outcomes** of Monte-Carlo tests (less variance is better).

Indeed the description is right, but the equation is wrong..will correct that.

]]>In my opinion Degrees of freedom decrease Statistical Significance .. not enhance

**Statistical significance ∝ ( Number of trades * degrees-of-freedom ) / Robustness test results variance**

In simple words this means that the **total statistical significance** (higher value is better) is proportional to the **total number of trades** (more is better) within your testing period and also proportional to **total amount of degrees-of-freedom** (input parameters = less is better), and inverse-proportional to the observed **variation of the outcomes** of Monte-Carlo tests (less variance is better).

Additionally I look for a reasonable RetDD number (>> 0) and reasonable DD, at the following confidence levels: 95%, 95% and 100%. Below an example of a good system.

]]>Moreover the results of some tests like randomization of data (random trades) contain very usefull info like the maximum expected DD level which you can use as a threshold to kill or not to kill your strategies in real trading.

gr

chris

Thank you for explaining. That does make sense as it can help to understand where it failed instead of just that it did fail.

I was doing MC last night. One thing I want to make sure of. With the number outputs (WF Net Profit, Drawdown, etc.), I understand that. You can use these numbers and compare to the backtest to determine its robustness.

But on the graph, what am I looking for? Am I looking for all the lines to be relatively the same as the main backtest's line? In other words, they are grouped together with very few or little outliers? So for example, the attached is an example of a bad one because the lines are not grouped tightly, right?

-Jonathan

]]>gr

chris

]]>Why do the MC's have to be done separately? Can it not be done in one MC test?

Yes it can all be done in one step, however I am also interested in strategies that did not pass each single MC step, I always try to understand why... so doing this in separate steps adds a little bit of additional diagnostics of the entire workflow.

So I do it because of informative reasons not functional. That’s how I learned it, but of course you should do what works the best for you.

greets

chris

]]>

There is no holy-grail in auto-trading. With all tools we use here, we can easily generate a strategy with a nice steep equity curve, that goes 'to-the-moon', but this will be pure curve-fitting. Instead we want to concentrate on the following two most important things: **statistical significance** and **system stability (robustness)**, and not the highest profits (as many of you would think). The main goal is to find as many as possible, stable and robust strategies that make some profits over time and add them to our portfolio of strategies.

Let's go back to our theory..let's create a perfect strategy with a WinRate of 80% (thick golden line) and compare it 500 randomly generated strategies (the figure below shows only first 100 random curves). Let's also print a histogram of all trades to see where the result of our strategy is placed in the probability distribution curve.

Figure 4.1: Results distribution of 100 trades

We can see that after 100 trades, our 80% WinRate strategy (golden line) if far from the 'random noise' dominated by the results of 500 random systems. The resulting blue bin of our golden line which is visible in our histogram is very small since the occurrence is only 1 (we have only one outcome of our winning system and 500 outcomes of randomly generated systems).

At this moment we see that our golden line result is statistical significant (since it beats all 500 random systems after 100 trades), but we still cannot tell if our result is curve-fitted or not. The only possible way to find it out is to perform several robustness tests on our golden line strategy. Here for we can do the following, e.g.:

- We can randomize the input parameters of our golden line strategy and see if this will have large influence on our results
- We can randomly skip trades within results to see if the equity curve will still be profitable
- We can randomly change the start date of our systems to see if the date change will have impact on the results
- We can randomly change market data to is if system will still perform well if the market will slightly change
- Many many more...

During our validation period we will 'jiggle' (= randomize) many different parameters around our strategy and strategy settings to check for robustness. This method is know as **Monte-Carlo **analysis method which is widely used in science. We will use that a lot since it is one of the most important stability tests.

**The perfect strategy:**

Let's imagine that our golden line strategy is perfect and none our Monte-Carlo tests has any influence on the final outcome. In that case the final result will be always the same, so for example after 50 different randomized sweeps, our probability histogram will look as follows:

Figure 4.2: Perfect strategy distribution

Now we can see, that our blue bin is much larger since all of 50 randomized sweeps result in the same outcome. This means that our strategy is not affected by any of 50 random tests we could think of. In scientific terms we could say that our Signal-to-Noise ratio is very high (Signal=our golden line results, Noise=random strategies). Our strategy is perfect!

However in real life, this will be very different, since no strategy is perfect and we will run much more randomized sweeps in our Monte-Carlo analysis. (StrategyQuant has several different robustness test modes which we will use extensively in our strategy validation). This all will result in a 'cloud' of golden lines, where each line represent outcome of different randomized sweep. Our probability histogram will also look very different, see the following figure:

Figure 4.3: Real strategy distribution

**CONCLUSION:**

Up to now we have seen that:

- We need to have strong statistical significance of our results (use high amount of trades)
- We need to prevent curve-fitting (decrease the degrees-of-freedom = use simple systems with less parameters)
- We need to pass all Monte-Carlo robustness tests with as low outcome variance as possible

Those three golden rules can be put into one beautiful equation that describes the resulting **total statistical significance**:

**Statistical significance ∝ Number of trades / (degrees-of-freedom * Robustness test results variance)**

In simple words this means that the **total statistical significance** (higher value is better) is proportional to the **total number of trades** (more is better) within your testing period and also proportional to **total amount of degrees-of-freedom** (input parameters = less is better), and inverse-proportional to the observed **variation of the outcomes** of Monte-Carlo tests (less variance is better).

HOWEVER: Statistical significance is only one critical part in algo-trading. There are much more characteristics a profitable system should have. In practice all those characteristics are validated using well designed strategy validation tests. So, to make this conclusion complete, a good strategy should pass at least the following additional validation tests:

**Strategy generation step:**In this step strategies are generated.**OOS1:**This test will run all strategies in an Out-Of-Sample period.**Slippage at 3:**Test at higher slippage of 3pips.**Different market:**Test on a different market.**A: Lower timeframe:**Test on a lower TimeFrame.**B: Higher timeframe:**Test on a higher TimeFrame.**MC random trades:**Monte-Carlo randomized trades test.**MC skipping trades:**Monte-Carlo random skipping of trades test.**MC random parameters:**Monte-Carlo randomized strategy parameters test.**MC random volatility ATR:**Monte-Carlo randomized market volatility test.**MC random slippage:**Monte-Carlo randomized slippage test.**MC random spread:**Monte-Carlo randomized spread test.**Last OOS2 test:**Final test using 1 year 'unseen' market data. This test simulates forward-test results.**Strategy selection:**Selection of the top best strategies (highest Ret/DD)**Portfolio correlation check:**Removing strongly correlated strategies**WFM:**Walk-Forward matrix optimization and strategy selection**Forward trading**

As we can see this means, that in order to validate 1 good strategy a huge amount of computation work is required. And for most of the people this level of programming skills is 'out of reach'. That is why we need good algo-trading tools. Tools that automate every single step required to generate, validate and identify all good strategies. On this forum everything we do is based on this one algorithmic trading tool: see next lesson.

Now we know what are the key elements we should be looking for during the strategy design period. You see that it is not easy, but totally possible..and we will tell you how: Read next lesson.

Gr

Chris

]]>