Lesson 4: What to l...

[Sticky] Lesson 4: What to look for in algo-trading?  


Member Admin
Joined: 3 months ago
Posts: 82
20/12/2018 11:32 am  

Let's try to put it all together. Hopefully after reading this lesson you will have a good idea what to look for in algo-trading.

There is no holy-grail in auto-trading. With all tools we use here, we can easily generate a strategy with a nice steep equity curve, that goes 'to-the-moon', but this will be pure curve-fitting. Instead we want to concentrate on the following two most important things: statistical significance and system stability (robustness), and not the highest profits (as many of you would think).  The main goal is to find as many as possible, stable and robust strategies that make some profits over time and add them to our portfolio of strategies.

Let's go back to our theory..let's create a perfect strategy with a WinRate of 80% (thick golden line) and compare it 500 randomly generated strategies (the figure below shows only first 100 random curves). Let's also print a histogram of all trades to see where the result of our strategy is placed in the probability distribution curve.

Figure 4.1: Results distribution of 100 trades

We can see that after 100 trades, our 80% WinRate strategy (golden line) if far from the 'random noise' dominated by the results of 500 random systems. The resulting blue bin of our golden line which is visible in our histogram is very small since the occurrence is only 1 (we have  only one outcome of our winning system and 500 outcomes of randomly generated systems).

At this moment we see that our golden line result is statistical significant (since it beats all 500 random systems after 100 trades), but we still cannot tell if our result is curve-fitted or not. The only possible way to find it out is to perform several robustness tests on our golden line strategy. Here for we can do the following, e.g.:

  • We can randomize the input parameters of our golden line strategy and see if this will have large influence on our results
  • We can randomly skip trades within results to see if the equity curve will still be profitable
  • We can randomly change the start date of our systems to see if the date change will have impact on the results
  • We can randomly change market data to is if system will still perform well if the market will slightly change
  • Many many more...

During our validation period we will 'jiggle' (= randomize) many different parameters around our strategy and strategy settings to check for robustness. This method is know as Monte-Carlo analysis method which is widely used in science. We will use that a lot since it is one of the most important stability tests.

The perfect strategy:

Let's imagine that our golden line strategy is perfect and none our Monte-Carlo tests has any influence on the final outcome. In that case the final result will be always the same, so for example after 50 different randomized sweeps, our probability histogram will look as follows:

Figure 4.2: Perfect strategy distribution

Now we can see, that our blue bin is much larger since all of 50 randomized sweeps result in the same outcome. This means that our strategy is not affected by any of 50 random tests we could think of. In scientific terms we could say that our Signal-to-Noise ratio is very high (Signal=our golden line results, Noise=random strategies). Our strategy is perfect!

However in real life, this will be very different, since no strategy is perfect and we will run much more randomized sweeps in our Monte-Carlo analysis. (StrategyQuant has several different robustness test modes which we will use extensively in our strategy validation). This all will result in a 'cloud' of golden lines, where each line represent outcome of different randomized sweep. Our probability histogram will also look very different, see the following figure:

Figure 4.3: Real strategy distribution


Up to now we have seen that:

  1. We need to have strong statistical significance of our results (use high amount of trades)
  2. We need to prevent curve-fitting (decrease the degrees-of-freedom = use simple systems with less parameters)
  3. We need to pass all Monte-Carlo robustness tests with as low outcome variance as possible

Those three golden rules can be put into one beautiful equation that describes the resulting total statistical significance:

Statistical significance ∝ ( Number of trades * degrees-of-freedom ) / Robustness test results variance

In simple words this means that the total statistical significance (higher value is better) is proportional to the total number of trades (more is better) within your testing period and also proportional to total amount of degrees-of-freedom (input parameters = less is better), and inverse-proportional to the observed variation of the outcomes of Monte-Carlo tests (less variance is better).

HOWEVER: Statistical significance is only one critical part in algo-trading. There are much more characteristics a profitable system should have. In practice all those characteristics are validated using well designed strategy validation tests. So, to make this conclusion complete, a good strategy should pass at least the following additional validation tests:

  • Strategy generation step: In this step strategies are generated.
  • OOS1: This test will run all strategies in an Out-Of-Sample period.
  • Slippage at 3: Test at higher slippage of 3pips.
  • Different market: Test on a different market.
  • A: Lower timeframe: Test on a lower TimeFrame.
    B: Higher timeframe: Test on a higher TimeFrame.
  • MC random trades: Monte-Carlo randomized trades test.
  • MC skipping trades: Monte-Carlo random skipping of trades test.
  • MC random parameters: Monte-Carlo randomized strategy parameters test.
  • MC random volatility ATR: Monte-Carlo randomized market volatility test.
  • MC random slippage: Monte-Carlo randomized slippage test.
  • MC random spread: Monte-Carlo randomized spread test.
  • Last OOS2 test: Final test using 1 year 'unseen' market data.  This test simulates forward-test results.
  • Strategy selection: Selection of the top best strategies (highest Ret/DD)
  • Portfolio correlation check: Removing strongly correlated strategies
  • WFM: Walk-Forward matrix optimization and strategy selection
  • Forward trading

As we can see this means, that in order to validate 1 good strategy a huge amount of computation work is required. And for most of the people this level of programming skills is 'out of reach'. That is why we need good algo-trading tools. Tools that automate every single step required to generate, validate and identify all good strategies. On this forum everything we do is based on this one algorithmic trading tool: see next lesson.

Now we know what are the key elements we should be looking for during the strategy design period. You see that it is not easy, but totally possible..and we will tell you how: Read next lesson.

This topic was modified 3 months ago 9 times by coensio
This topic was modified 1 month ago by coensio

glennsia and valesch liked

Please Login or Register