Lesson 3: How to pr...
 

[Sticky] Lesson 3: How to prevent curve-fitting.  

  RSS

coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
19/12/2018 10:38 am  

Curve-fitting will kill any good system. At first you need to understand what curve-fitting means, it simply means: connecting ‘the dots’. In case of trading. ‘the dots’ are 'randomly' distributed results on the equity curve. During optimization the strategy tester connects the dots in several different ways (thousands of different ways, depending on the amount of available parameters and their optimization range = 'degrees-of-freedom'). See the following figures:

Figure 3.1: Example of curve fitting

Since curve-fitting is a mathematical process it can be minimized by the following two approaches:

  • Minimize the number of parameters: Each new EA parameter will increase the total resulting degrees of freedom in which the resulting equity line can be ‘bend’ or in other words, it will increase the number of ways in which the dots can be connected. With only one parameter which has only two settings ‘A’ or ‘B’ the total number in degrees of freedom is 2. This results in one straight line. The more parameter the more EA settings possibilities the bigger chance of curve-fitting. Thus, one way to fight curve-fitting is decrease the complexity of trading system by minimizing its parameters and range of parameters.
  • Increasing the number of trades: Following the logic presented above, the more trades which are used to optimize given trading system, the more difficult it will become to curve-fit the resulting equity line. Results based on a high amount of trades tend to be more stable and have less chance to be curve-fitted.
  • (New) OOS: Test your strategies on a long OOS period (out-of-sample), doing so you can test your strategy on 'unseen data', the longer the OOS window the better, if the performance of your strategy is still OK when tested with unseen data, then your strategy is probably not curve-fitted.
  • (New) Cross market: Another way to test if your strategy is not curve-fitted is by testing it on an another (but correlated) instrument. E.g.: after developing a strategy on EURUSD you can test it also on a GBPUSD pair (which has very strong correlation to EURUSD), if your strategy performs still reasonable on this another pair then it is probably not curve-fitted to EURUSD data.   

 

Conclusion: To prevent curve-fitting, we need to reduce complexity (degrees-of-freedom) and optimize using high amount of trades (See lesson 2!).

 

Following the rules of probability which are explained in the previous lessons, we can conclude that the requirements for stable system design are quite contradicting, namely:

  • On one side we need to remove complexity of our systems to fight curve-fitting BUT on the other hand we need to add complexity in order to be able to search for profitable strategies.
  • Adding complexity to our trading systems (by increasing the number of indicators and rules) leads to high curve-fitting risk, and also to 'over-sepecialization' --> systems with too many trading rules (large amount of parameters = degrees-of-freedom) will be able to pick-up only the top high quality signals --> this will dramatically limit the number of trades in your optimization/backtest! Making your results less reliable. See lesson 2.

 

Solution: KEEP IT SIMPLE!

Gr
Chris
This topic was modified 5 months ago 6 times by coensio
This topic was modified 2 months ago by coensio

Quote
BladeRunner
(@bladerunner)
Active Member
Joined: 5 months ago
Posts: 11
20/12/2018 10:44 pm  

As a multi-year SQ user, I think we will have lots of good stuff to discuss here. But first I'll have to find time to go through your posts, which reflect your personal experience and usage of SQ.


ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
21/12/2018 8:18 am  

Great to here that we have also experienced users here...personally I propose to concentrate on the new SQ-X platform. 


ReplyQuote
BladeRunner
(@bladerunner)
Active Member
Joined: 5 months ago
Posts: 11
22/12/2018 5:51 pm  
Posted by: coensio

Great to here that we have also experienced users here...personally I propose to concentrate on the new SQ-X platform. 

Yes, you are right. I've been resisting to change as my several attempts to move to SQ X (or the release candidates) where not very productive but at a certain point one has to move on.


ReplyQuote
BladeRunner
(@bladerunner)
Active Member
Joined: 5 months ago
Posts: 11
22/12/2018 6:08 pm  
Posted by: coensio

Curve-fitting will kill any good system. At first you need to understand what curve-fitting means, it simply means: connecting ‘the dots’. In case of trading. ‘the dots’ are 'randomly' distributed results on the equity curve. During optimization the strategy tester connects the dots in several different ways (thousands of different ways, depending on the amount of available parameters and their optimization range = 'degrees-of-freedom'). See the following figures:

Figure 3.1: Example of curve fitting

Since curve-fitting is a mathematical process it can be minimized by the following two approaches:

  • Minimize the number of parameters: Each new EA parameter will increase the total resulting degrees of freedom in which the resulting equity line can be ‘bend’ or in other words, it will increase the number of ways in which the dots can be connected. With only one parameter which has only two settings ‘A’ or ‘B’ the total number in degrees of freedom is 2. This results in one straight line. The more parameter the more EA settings possibilities the bigger chance of curve-fitting. Thus, one way to fight curve-fitting is decrease the complexity of trading system by minimizing its parameters and range of parameters.
  • Increasing the number of trades: Following the logic presented above, the more trades which are used to optimize given trading system, the more difficult it will become to curve-fit the resulting equity line. Results based on a high amount of trades tend to be more stable and have less chance to be curve-fitted.

 

Conclusion: To prevent curve-fitting, we need to reduce complexity (degrees-of-freedom) and optimize using high amount of trades (See lesson 2!).

 

Following the rules of probability which are explained in the previous lessons, we can conclude that the requirements for stable system design are quite contradicting, namely:

  • On one side we need to remove complexity of our systems to fight curve-fitting BUT on the other hand we need to add complexity in order to be able to search for profitable strategies.
  • Adding complexity to our trading systems (by increasing the number of indicators and rules) leads to high curve-fitting risk, and also to 'over-sepecialization' --> systems with too many trading rules (large amount of parameters = degrees-of-freedom) will be able to pick-up only the top high quality signals --> this will dramatically limit the number of trades in your optimization/backtest! Making your results less reliable. See lesson 2.

 

Solution: KEEP IT SIMPLE!

Gr
Chris

This is interesting and I would subscribe them totally together with a couple of other possible measures.  But my question is, how many trades do you consider to be sufficient to avoid too much curve fitting (since there will always be a degree of it ? My most interesting strategies developed with SQ all have around 500 to 700 trades for 10+ years and you posted a strategy in another thread with around 300 trades for a 15 year period.

From my "gut feeling" and generally around 300 trades would be too few to avoid curve fitting, 500 would be in the border of acceptable and 700 would be where you would probably have some kind of psychological comfort. The "gut feeling" also comes from other statistics cases (not transferable to this case) where a sample of dimension= 300 might not be statistically relevant or too small. Thanks for your input. 

 

 

This post was modified 5 months ago 2 times by BladeRunner

ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
22/12/2018 6:46 pm  

BladeRunner

You need to read lesson 4. Basically, it all comes down to having a strong statistical significance of your results and this is very hard to achieve, this is because we can see that a simple system with only one degree-of-freedom (one parameter) needs more than 50...100 trades to show strong significance. Most systems have more than one degree-of-freedom because they have multiple input parameters that can be optimized. So pragmatically you can say that your X-number (the minimum required number of trades) is equal to 50...100 * number of parameters in your system. In most cases you will not get that many trades. So we need to concentrate on the results from our stability tests (Monte-Carlo) to validate our statistical significance.

In addition I must add that lucky for us this applies only for IS data, in most cases for OOS/forward-tests data, 50..100 trades is enough to tell if the system is profitable or not.

It is hard to achieve even if you know the theoretical background, auto-trading is like looking for a needle in a haystack. That's why I need help, that is the main reason behind this forum. I share my knowledge for free in hope you will help me find as many needles as possible;)

I hope it helps, see lesson 4.

Gr

Chris

www.coensio.com

 

This post was modified 5 months ago by coensio

ReplyQuote
BladeRunner
(@bladerunner)
Active Member
Joined: 5 months ago
Posts: 11
22/12/2018 7:20 pm  
Posted by: coensio

BladeRunner

You need to read lesson 4. Basically, it all comes down to having a strong statistical significance of your results and this is very hard to achieve, this is because we can see that a simple system with only one degree-of-freedom (one parameter) needs more than 50...100 trades to show strong significance. Most systems have more than one degree-of-freedom because they have multiple input parameters that can be optimized. So pragmatically you can say that your X-number (the minimum required number of trades) is equal to 50...100 * number of parameters in your system. In most cases you will not get that many trades. So we need to concentrate on the results from our stability tests (Monte-Carlo) to validate our statistical significance.

In addition I must add that lucky for us this applies only for IS data, in most cases for OOS/forward-tests data, 50..100 trades is enough to tell if the system is profitable or not.

It is hard to achieve even if you know the theoretical background, auto-trading is like looking for a needle in a haystack. That's why I need help, that is the main reason behind this forum. I share my knowledge for free in hope you will help me find as many needles as possible;)

I hope it helps, see lesson 4.

Gr

Chris

www.coensio.com

 

Thanks. I had already peaked in lesson 4 because this topic interest me (actually I think it's quite decisive), but I will see it in detail. Thanks.

PS - Can you detail what you mean with "That's why I need help, that is the main reason behind this forum. I share my knowledge for free in hope you will help me find as many needles as possible;)" . What are your expectations of what this community can help you with?


ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
22/12/2018 7:31 pm  

Expectations:

1. Let's try to find as many 'interesting' strategies as possible and run monitored forward tests on a live account (I will risk my money and use my monitoring tools for it).

2. Let's once for all define a stable workflow (design process) that increases chances of finding profitable strategies. This is something that is never defined well by SQ team, there are too many possibilities. For example, which Monte-Carlo tests are valuable and which are not, how to pre-filter outcomes during strategy generation? Which generation method to use to begin with etc.. If I look at the new SQ-X I am dawning in amount of options...they made things only more complex, don't you think?


ReplyQuote
maegop
(@maegop)
New Member
Joined: 4 months ago
Posts: 3
10/01/2019 8:48 pm  

I totally agree with your opinion, there are a huge amount of possibilities in SQ X, and they do not have recommended parameters for their tool. It will be very useful that in this forum people could help to define the values with they have certain results. It will take time and effort but if people support your idea we will have good results. Alone we will not go anywhere.

Thanks @coensio for your great initiative, idea, forum, and knowledge sharing.


ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
11/01/2019 7:59 am  
Posted by: maegop

I totally agree with your opinion, there are a huge amount of possibilities in SQ X, and they do not have recommended parameters for their tool. It will be very useful that in this forum people could help to define the values with they have certain results. It will take time and effort but if people support your idea we will have good results. Alone we will not go anywhere.

Thanks @coensio for your great initiative, idea, forum, and knowledge sharing.

Indeed that was my main idea behind this forum. The number of possibilities in the new SQ-X seems to be higher than the number of stars in our visible universe 😉 We, as community need to reach a consensus w.r.t. the proper settings and proper workflow. As we speak I'm spending many hours a day to test all possibilities. I hope you will do the same so we can share the results.

Gr

Chris

www.coensio.com


ReplyQuote
Share:
  
Working

Please Login or Register