# Forum

Lesson 3: How to pr...

Clear all

# [Sticky] Lesson 3: How to prevent curve-fitting.

Page 1 / 2
(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter

Curve-fitting will kill any good system. At first you need to understand what curve-fitting means, it simply means: connecting ‘the dots’. In case of trading. ‘the dots’ are 'randomly' distributed results on the equity curve. During optimization the strategy tester connects the dots in several different ways (thousands of different ways, depending on the amount of available parameters and their optimization range = 'degrees-of-freedom'). See the following figures:

Figure 3.1: Example of curve fitting

Since curve-fitting is a mathematical process it can be minimized by the following two approaches:

• Minimize the number of parameters: Each new EA parameter will increase the total resulting degrees of freedom in which the resulting equity line can be ‘bend’ or in other words, it will increase the number of ways in which the dots can be connected. With only one parameter which has only two settings ‘A’ or ‘B’ the total number in degrees of freedom is 2. This results in one straight line. The more parameter the more EA settings possibilities the bigger chance of curve-fitting. Thus, one way to fight curve-fitting is decrease the complexity of trading system by minimizing its parameters and range of parameters.
• Increasing the number of trades: Following the logic presented above, the more trades which are used to optimize given trading system, the more difficult it will become to curve-fit the resulting equity line. Results based on a high amount of trades tend to be more stable and have less chance to be curve-fitted.
• (New) OOS: Test your strategies on a long OOS period (out-of-sample), doing so you can test your strategy on 'unseen data', the longer the OOS window the better, if the performance of your strategy is still OK when tested with unseen data, then your strategy is probably not curve-fitted.
• (New) Cross market: Another way to test if your strategy is not curve-fitted is by testing it on an another (but correlated) instrument. E.g.: after developing a strategy on EURUSD you can test it also on a GBPUSD pair (which has very strong correlation to EURUSD), if your strategy performs still reasonable on this another pair then it is probably not curve-fitted to EURUSD data.

Conclusion: To prevent curve-fitting, we need to reduce complexity (degrees-of-freedom) and optimize using high amount of trades (See lesson 2!).

Following the rules of probability which are explained in the previous lessons, we can conclude that the requirements for stable system design are quite contradicting, namely:

• On one side we need to remove complexity of our systems to fight curve-fitting BUT on the other hand we need to add complexity in order to be able to search for profitable strategies.
• Adding complexity to our trading systems (by increasing the number of indicators and rules) leads to high curve-fitting risk, and also to 'over-sepecialization' --> systems with too many trading rules (large amount of parameters = degrees-of-freedom) will be able to pick-up only the top high quality signals --> this will dramatically limit the number of trades in your optimization/backtest! Making your results less reliable. See lesson 2.

Solution: KEEP IT SIMPLE!

Gr
Chris
This topic was modified 3 years ago 6 times by coensio
This topic was modified 2 years ago by coensio

Active Member Customer
Joined: 3 years ago
Posts: 19

As a multi-year SQ user, I think we will have lots of good stuff to discuss here. But first I'll have to find time to go through your posts, which reflect your personal experience and usage of SQ.

(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter

Great to here that we have also experienced users here...personally I propose to concentrate on the new SQ-X platform.

Active Member Customer
Joined: 3 years ago
Posts: 19

Posted by: coensio

Great to here that we have also experienced users here...personally I propose to concentrate on the new SQ-X platform.

Yes, you are right. I've been resisting to change as my several attempts to move to SQ X (or the release candidates) where not very productive but at a certain point one has to move on.

Active Member Customer
Joined: 3 years ago
Posts: 19

Posted by: coensio

Curve-fitting will kill any good system. At first you need to understand what curve-fitting means, it simply means: connecting ‘the dots’. In case of trading. ‘the dots’ are 'randomly' distributed results on the equity curve. During optimization the strategy tester connects the dots in several different ways (thousands of different ways, depending on the amount of available parameters and their optimization range = 'degrees-of-freedom'). See the following figures:

Figure 3.1: Example of curve fitting

Since curve-fitting is a mathematical process it can be minimized by the following two approaches:

• Minimize the number of parameters: Each new EA parameter will increase the total resulting degrees of freedom in which the resulting equity line can be ‘bend’ or in other words, it will increase the number of ways in which the dots can be connected. With only one parameter which has only two settings ‘A’ or ‘B’ the total number in degrees of freedom is 2. This results in one straight line. The more parameter the more EA settings possibilities the bigger chance of curve-fitting. Thus, one way to fight curve-fitting is decrease the complexity of trading system by minimizing its parameters and range of parameters.
• Increasing the number of trades: Following the logic presented above, the more trades which are used to optimize given trading system, the more difficult it will become to curve-fit the resulting equity line. Results based on a high amount of trades tend to be more stable and have less chance to be curve-fitted.

Conclusion: To prevent curve-fitting, we need to reduce complexity (degrees-of-freedom) and optimize using high amount of trades (See lesson 2!).

Following the rules of probability which are explained in the previous lessons, we can conclude that the requirements for stable system design are quite contradicting, namely:

• On one side we need to remove complexity of our systems to fight curve-fitting BUT on the other hand we need to add complexity in order to be able to search for profitable strategies.
• Adding complexity to our trading systems (by increasing the number of indicators and rules) leads to high curve-fitting risk, and also to 'over-sepecialization' --> systems with too many trading rules (large amount of parameters = degrees-of-freedom) will be able to pick-up only the top high quality signals --> this will dramatically limit the number of trades in your optimization/backtest! Making your results less reliable. See lesson 2.

Solution: KEEP IT SIMPLE!

Gr
Chris

This is interesting and I would subscribe them totally together with a couple of other possible measures.  But my question is, how many trades do you consider to be sufficient to avoid too much curve fitting (since there will always be a degree of it ? My most interesting strategies developed with SQ all have around 500 to 700 trades for 10+ years and you posted a strategy in another thread with around 300 trades for a 15 year period.

From my "gut feeling" and generally around 300 trades would be too few to avoid curve fitting, 500 would be in the border of acceptable and 700 would be where you would probably have some kind of psychological comfort. The "gut feeling" also comes from other statistics cases (not transferable to this case) where a sample of dimension= 300 might not be statistically relevant or too small. Thanks for your input.

This post was modified 3 years ago 2 times by BladeRunner

(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter

You need to read lesson 4. Basically, it all comes down to having a strong statistical significance of your results and this is very hard to achieve, this is because we can see that a simple system with only one degree-of-freedom (one parameter) needs more than 50...100 trades to show strong significance. Most systems have more than one degree-of-freedom because they have multiple input parameters that can be optimized. So pragmatically you can say that your X-number (the minimum required number of trades) is equal to 50...100 * number of parameters in your system. In most cases you will not get that many trades. So we need to concentrate on the results from our stability tests (Monte-Carlo) to validate our statistical significance.

In addition I must add that lucky for us this applies only for IS data, in most cases for OOS/forward-tests data, 50..100 trades is enough to tell if the system is profitable or not.

It is hard to achieve even if you know the theoretical background, auto-trading is like looking for a needle in a haystack. That's why I need help, that is the main reason behind this forum. I share my knowledge for free in hope you will help me find as many needles as possible;)

I hope it helps, see lesson 4.

Gr

Chris

www.coensio.com

This post was modified 3 years ago by coensio

Active Member Customer
Joined: 3 years ago
Posts: 19

Posted by: coensio

You need to read lesson 4. Basically, it all comes down to having a strong statistical significance of your results and this is very hard to achieve, this is because we can see that a simple system with only one degree-of-freedom (one parameter) needs more than 50...100 trades to show strong significance. Most systems have more than one degree-of-freedom because they have multiple input parameters that can be optimized. So pragmatically you can say that your X-number (the minimum required number of trades) is equal to 50...100 * number of parameters in your system. In most cases you will not get that many trades. So we need to concentrate on the results from our stability tests (Monte-Carlo) to validate our statistical significance.

In addition I must add that lucky for us this applies only for IS data, in most cases for OOS/forward-tests data, 50..100 trades is enough to tell if the system is profitable or not.

It is hard to achieve even if you know the theoretical background, auto-trading is like looking for a needle in a haystack. That's why I need help, that is the main reason behind this forum. I share my knowledge for free in hope you will help me find as many needles as possible;)

I hope it helps, see lesson 4.

Gr

Chris

www.coensio.com

Thanks. I had already peaked in lesson 4 because this topic interest me (actually I think it's quite decisive), but I will see it in detail. Thanks.

PS - Can you detail what you mean with "That's why I need help, that is the main reason behind this forum. I share my knowledge for free in hope you will help me find as many needles as possible;)" . What are your expectations of what this community can help you with?

(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter

Expectations:

1. Let's try to find as many 'interesting' strategies as possible and run monitored forward tests on a live account (I will risk my money and use my monitoring tools for it).

2. Let's once for all define a stable workflow (design process) that increases chances of finding profitable strategies. This is something that is never defined well by SQ team, there are too many possibilities. For example, which Monte-Carlo tests are valuable and which are not, how to pre-filter outcomes during strategy generation? Which generation method to use to begin with etc.. If I look at the new SQ-X I am dawning in amount of options...they made things only more complex, don't you think?

(@maegop)
New Member
Joined: 3 years ago
Posts: 3

I totally agree with your opinion, there are a huge amount of possibilities in SQ X, and they do not have recommended parameters for their tool. It will be very useful that in this forum people could help to define the values with they have certain results. It will take time and effort but if people support your idea we will have good results. Alone we will not go anywhere.

Thanks @coensio for your great initiative, idea, forum, and knowledge sharing.

(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter
Posted by: maegop

I totally agree with your opinion, there are a huge amount of possibilities in SQ X, and they do not have recommended parameters for their tool. It will be very useful that in this forum people could help to define the values with they have certain results. It will take time and effort but if people support your idea we will have good results. Alone we will not go anywhere.

Thanks @coensio for your great initiative, idea, forum, and knowledge sharing.

Indeed that was my main idea behind this forum. The number of possibilities in the new SQ-X seems to be higher than the number of stars in our visible universe 😉 We, as community need to reach a consensus w.r.t. the proper settings and proper workflow. As we speak I'm spending many hours a day to test all possibilities. I hope you will do the same so we can share the results.

Gr

Chris

www.coensio.com

(@denistars)
New Member
Joined: 2 years ago
Posts: 1

Good morning.
I agree and I like the idea of ​​working in a team to find the most profitable strategies.
I have been using sqx for months and am studying and doing a lot of tests.

I have been programming mql4 for 4 years and I dedicate several hours to it.

I congratulate you for this site.

If you have a workflow to run, I am available.

(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter

Unfortunately our collaboration project is as good as dead :/ 30 people signed up only few collaborated....this is pointless...most people joined just to get free strategies.

(@davidyan)
New Member
Joined: 2 years ago
Posts: 2

@coensio Can't agree more with your idea of SFT. It is very easy to find strategies that works well on all IS and OOS data, and pass all MC tests, but when turn to those data that not included in IS and OOS, most of the strategies fail. This means the strategies produced by the workflow does not work in the future even if they have nice equity curve in all history data.
Now I'm wondering how to solve this problem. It is not a problem to train a "good" strategy model that fit the history data well, it is a problem to find a "workflow" that produce stable strategy. SQ only solve the previous problem.
@coensio Is it possible to join your collaboration group now? I really eager to help each other. I'm not to ask for a free strategy, but for learning new ideas.

(@coensio)
Joined: 3 years ago
Posts: 191
Topic starter

Yes that's the main issue with the workflows...they MUST generate some market edge (on average) using the "future/past" OOS data otherwise they are worthless!

From my experience: the only workflow I could generate that REPEATABLY generated profits during "future/past" OOSses, outside the workflow data is on futures markets. In Forex it's like being Alice in Wonderland 😉

The collaboration has been stopped..since most of people were not collaborating at all...

(@smarty)
New Member
Joined: 1 year ago
Posts: 3

@coensio

Hi Coensio, First of all, I congratulate you on the initiative of this forum. I also understand why collaborations are interrupted, it is difficult to find who can tame the complexity of SQX. About SQX I have some key questions, I would be grateful if you answer me.

I have been observing SQX for 2 weeks, take into account that I am a software engineer, and that for about 5 years I have been "playing" with the development of strategies and indicators, software development is not a problem for me. For example, I know Java and I could potentially develop snippets for extend SQX.

I have to decide whether to purchase the SQX.

I would like to know if, in your opinion, it is right to invest time and money in the SQX product, in order to be able to set up a reasonable profitable trading. I repeat, I am speaking about potential.

And, to better explain the distinction you made between approaching futures and forex markets.

Thank you for attention, cordiality, Sabino.

Page 1 / 2
Share: