Statistical significance is another huge reason why most traders fail in their auto-trading. Why? Because nobody talks about it...and I really do not know why...Statistical significance tell you one extremely important thing: if you can 'trust' your results observed in backtest or forward test or is everything what you see just due to random luck.
To make clear what statistical significance is let's look at the scientific concept know as P-Value. I will try to explain that using two examples:
This simple test also gives you the X number. This number is the minimum required number of coin tosses (or FX trades), needed to be able to tell if a given system (or an EA setting) has statistical edge or statistical significance. So, if you see a profitable setting after only 25 trades during optimization, you need to compare this to at least 19 other random trading systems (random coin tosses). If one or more random results produces equal or better results than your optimized system, then it is NOT a statistically significant result. That is why it is almost impossible to optimize using short term data (like: weekly basis), since each EA setting will produce not more than 25 trades. When comparing to random systems those random systems will always produce similar or even better results! You will not know if your system is profitable or if the positive result during optimization is caused by a random lucky shot.
You can test it by yourself, but the minimum valid number of trades >= 50. Only after 50 trades there will be a significant difference between all random systems and any profitable setting. 50 is the absolute minimum 150 or more is considered as stable (this number depends on 'degrees-of-freedom'...keep reading...). See the following example.
Figure 1: 25 trades = example of poor P-VALUE
As you can see in the example above after 25 trades the main strategy result (gold line) is not much better than randomly distributed results (based on random entry strategy, coin toss). In that case you can not say if this result is due to good strategy or just pure luck like show by random trading systems. This also means that the result is within 'first sigma' of probability distribution, among pure randomness.
Figure 4: 200 trades = example of good P-VALUE
Conclusion: In this example the selected strategy (EA setting) is profitable over long term and results in a strong P-Value of 5% (since the final result beats 19 random strategies).
Thus, in order to be able to say if the given setting is profitable or not we need to test it over a long(er) period of time using high amount of trades! The optimization/backtesting results based on a (too) small amount of trades (<100) have very low STATISTICAL SIGNIFICANCE and cannot be trusted!