Simulated Forward T...
 

Simulated Forward Test method  

  RSS

coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
25/02/2019 11:06 am  

Recently I introduced a simple method that allowed me to verify and improve my current strategy design workflow, you can read it on my blog: Simulated Forward Test method

Gr

Chris


Quote
sydap2003
(@sydap2003)
New Member
Joined: 5 months ago
Posts: 2
26/02/2019 5:23 am  

Hi Chris

Not sure this is all that new. Useful process if you want to to see how often and over what period one might want to re-optimise or re-strategise.

But why not just do the strategy development and optimisation over the most recent 10 year period. Ten years provides a reasonable amount of time over which market dynamics can swing back and forth or further diverge. Do the development/ testing/ optimisation with fixed lot size so that it is easy to see how performance varies over the 10 years. If a strategy can survive 10 yrs of market churn, there is a reasonable chance it will be able to handle conditions going forward. (3 to 4 yrs is far too short and gives misleading hope).

With this approach I have six strategies that show continual growth over the 10 yrs. With many others I find I can either get the strategies to perform well in the period up to around 2014 and then stagnate; or stagnate up to 2014 and then perform well. I would love to know what happened to market dynamics around 2013-2014 to cause this effect.

(I believe that tick data prior to 2008 is suspect and also skewed by the GFC; so 2009 is my earliest year for starting.)


ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
26/02/2019 1:36 pm  

Hi ,

I never said I invented this method, just that I forced my self  to use it. Apparently many traders are working in similar way only nobody is sharing this method with new traders...

There is a big misconception w.r.t. this SFT method. Most people thing it is meant ONLY to design/select strategies however my main goal is to validate my whole strategy design and selection process (and strategies with it). So I check if my process was also effective in previous years like 2014 or 2016....if it was not then it's a red flag form me.

To be honest I also think this is the only way to verify if you are a continuously successful with your system trading.

Take for example WFM: if you perform WFM using all available data up to now (2019), the results for previous years will be (most probably) different, than when you do it sequentially by pretending you start in the past e.g.: start in 2014 -> do your WFM-> test results on 2015 using best WFM settings -> do your WFM using data up to and including 2015 -> test results on 2016 using best WFM settings -> do your WFM using data up to and including 2016 -> etc.... This is the real flow of actions you would perform when you would start trading your strategies back in 2014.

But that's only my opinion, of course everybody should do what works best for them...There are so many gurus and so many opinions... 🙂

I like to see things in action, with my own eyes.

Gr

Chris

 


ReplyQuote
sydap2003
(@sydap2003)
New Member
Joined: 5 months ago
Posts: 2
26/02/2019 11:01 pm  

There sure are many opinions. And so it is worthwhile to share our thoughts as you have done. So thank you.

I guess my main concern in whatever way we do this, is to do one's development work over a period in which your data set of past market behaviour contains a reasonable degree of variability. For me the robustness of a strategy we develop is largely dependent on what sort of data set we use to develop it. Using a data set with limited variability, eg one or two years, does not result in a robust strategy no matter what development process we use.

But that is just my red flag experience.


ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
27/02/2019 8:53 am  
Posted by: sydap2003

There sure are many opinions. And so it is worthwhile to share our thoughts as you have done. So thank you.

I guess my main concern in whatever way we do this, is to do one's development work over a period in which your data set of past market behaviour contains a reasonable degree of variability. For me the robustness of a strategy we develop is largely dependent on what sort of data set we use to develop it. Using a data set with limited variability, eg one or two years, does not result in a robust strategy no matter what development process we use.

But that is just my red flag experience.

This is a different topic, but for me it is important to use a period long enough to capture at least 6 different types of market: bullish/bearish/stagnating and rising/falling/stagnating volatility. So you have a matrix of 3x3. In most cases this requires to develop systems using data from multiple years (10 years). If you want to do it even better then use multiple years and also multiple markets.

 


ReplyQuote
Cornus
(@cornus)
Active Member
Joined: 2 months ago
Posts: 5
24/03/2019 8:41 am  

Hi Chris,
I try to use the same method which you call SFT (I call it creation of tested workflows). I started this approach few weeks ago before finding your website and I am glad that I am not alone who uses this methodology. In my opinion, this approach is necessary for statistically sound generation of automatic strategies.

The only disadvantage of this method is its time demandingness. According to my experiences, it is very time-consuming to find a workflow with a reasonably high success rate on several OOS. However, I am still on the beginning and I hope that I will be able to find reliable workflows for several currency pairs.

You write on your blog that you developed a 100% accurate workflow. For which currency pair was the workflow developed? And how many tests are incorporated in this workflow? I hope that my questions are not too intrusive and I will understand if you do not reveal this information.

Best,
Petr

 

This post was modified 2 months ago by Cornus

ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
24/03/2019 9:49 am  

Hi Petr,

Recently this idea of SFT (or paper trading OOS), was rejected by some of the SQ (main)forum pseudo-gurus 😉 So I reached out to several professional traders, who are actually making money with trading. Their ALL use something similar and nobody calls this period just 'OOS', because it has this special purpose: to speed up system-to-market introduction. Normally, systems would be live tested (for several months) using a small real account, to see how they are performing in real trading conditions, so called: 'incubating period', before trading the systems on the main real account with much bigger lotsizes. However, many experienced traders, verify their systems and workflows, just by running them on this special reserved OOS period, which I call SFT and so far, I have seen the following naming: "paper trading OOS", "incubator OOS", "Super OOS", "paper trading period", etc..

So yes this idea is widely used to verify systems and workflows, however there are some issues with it:

- The real trading results can still be 'slightly' off, backtesting will never represent real results, but if you want to save time and you accept this 'mismatch', SFT is still a very good indication of system performance.

- Sometimes, reserving only 1 year of SFT (paper trading OOS), could be not enough to make a proper judgement of the real system performance. Example: when a system in this special OOS will be considered as PASS or FAILED? When profit is > $0? Or when DD is lower than historical DD? Or when stagnation is longer than expected...etc etc..? So this choice of pass/fail criteria will influence the resulting outcome. For example: if I use 1 year of SFT, and assume that system is consider as FAILED only when the Max DD has exceeded the previously observed historical Max DD, then yes, I have seen 100% accurate workflow(years). If I state that system is failed if the profit in SFT is  <$0, then the total score is between 65% to 100% depending on the specific workflow-year.

- Another funny observation on my side: if the total profit for a given system in SFT is < $0 and its DD in this period in almost at max as observed on historical data, is this a valid reason to eliminate this system or maybe a very good entry point to start trading this system on live account 😉 I really know some professional traders, that would say: trade it live! Because they never have more than 30% of all newly introduced system at their 'equity-high'.

The example I gave on the SQ-forum and my blog is based on EURUSD H1 and the final score is based on 100 systems.

Greets,

Chris

This post was modified 2 months ago 2 times by coensio

ReplyQuote
Cornus
(@cornus)
Active Member
Joined: 2 months ago
Posts: 5
24/03/2019 2:20 pm  

Hi Chris,
Thanks for your quick reply. I admit I don't read SQ forum too often, so that I missed your contributions. I think that this approach not only speeds up system-to-market introduction but more important, it estimates the chance that implemented strategies will be profitable in a given time horizon.

I fully agree that the real trading results will be somewhat off. This is inevitable due to many influences to real trading.

As for DD in almost at the observed maximum in backtest, I am not so brave to put such strategies on live account 🙂 And I don't believe that such approach is profitable, but I can be mistaken.

For the time being, I use two SFT time periods with 1-year OOS (1st SFT data = 2008 - 2016 and 1st SFT OOS = 2017; 2nd SFT data = 2009 - 2017 and 2nd SFT OOS = 2018). It seems to me that it is difficult to achieve a high success rate on 2-year OOS. Of course, the 1-year OOS means that each strategy should be terminated 1 year after its usage in the real trading which is a practical disadvantage.

My main criterion for SFT OOS is profit > 0 and my goal is to achieve the success rate at least 75%, so that at least 75% of implemented strategies should be profitable in the 1-year horizon. I also measure the relative frequency of Return/DD > 0.5, but I don't use this condition as the primary criterion. I have spent a lot of time with XAUUSD but my present results are not fully satisfactory yet.

Thanks very much for information about the currency pair and the interval of the total score. I will focus on EURUSD H1 now and, if I find some reasonable workflow (the total score at least 75%), I will send you my results so that we can discuss and compare our robustness tests and builder's settings.

Best,
Petr

This post was modified 2 months ago 2 times by Cornus

ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
24/03/2019 4:35 pm  

As for DD in almost at the observed maximum in backtest, I am not so brave to put such strategies on live account 🙂 And I don't believe that such approach is profitable, but I can be mistaken.

This really can be a workable approach, the assumption is that when system has reached its maximum expected DD during this SFT OOS, then its equity can only go up from there;) Following this assumption, you could put your ultimate strategy exit level using a very tight distance. Imagine a system has reached -$1000 DD in SFT OOS, and you know from your testing and MC tests that the maximum expected DD at 100% confidence is -$1500, then you could start trading this system with a limit of -$500 (risking only $500 from this point moving forward).

For the time being, I use two SFT time periods with 1-year OOS (1st SFT data = 2008 - 2016 and 1st SFT OOS = 2017; 2nd SFT data = 2009 - 2017 and 2nd SFT OOS = 2018). It seems to me that it is difficult to achieve a high success rate on 2-year OOS. Of course, the 1-year OOS means that each strategy should be terminated 1 year after its usage in the real trading which is a practical disadvantage.

Yes, indeed if your systems cannot be re-optimized using WFA, then in most cases they will 'run out of steam' X months/years after generation period. We cannot expect systems will work forever...nothing does...

My main criterion for SFT OOS is profit > 0 and my goal is to achieve the success rate at least 75%, so that at least 75% of implemented strategies should be profitable in the 1-year horizon. I also measure the relative frequency of Return/DD > 0.5, but I don't use this condition as the primary criterion. I have spent a lot of time with XAUUSD but my present results are not fully satisfactory yet.

Last thing I can add is, that in most cases the generated systems can be/will be somehow correlated with each other...so this can give a false view of your total success rate...in the end my personal goal is to select only few uncorrelated systems for live trading, using a workflow that has been proven and was able to produce profitable systems in the past.

 


ReplyQuote
Cornus
(@cornus)
Active Member
Joined: 2 months ago
Posts: 5
24/03/2019 7:09 pm  

This really can be a workable approach, the assumption is that when system has reached its maximum expected DD during this SFT OOS, then its equity can only go up from there;) Following this assumption, you could put your ultimate strategy exit level using a very tight distance. Imagine a system has reached -$1000 DD in SFT OOS, and you know from your testing and MC tests that the maximum expected DD at 100% confidence is -$1500, then you could start trading this system with a limit of -$500 (risking only $500 from this point moving forward).

Maybe you are right but it'd be nice to see a test of this hypothesis. I mean to take a set of such strategies and to examine how many of them were profitable in an extra OOS which follows the SFT OOS.

Last thing I can add is, that in most cases the generated systems can be/will be somehow correlated with each other...so this can give a false view of your total success rate...in the end my personal goal is to select only few uncorrelated systems for live trading, using a workflow that has been proven and was able to produce profitable systems in the past.

I agree, the success rate can be distorted by correlated strategies. I sometime implement correlated strategies but I proportionally decrease their trading volumes (I use fixed size money management).


ReplyQuote
sam
 sam
(@sam)
New Member
Joined: 3 months ago
Posts: 1
30/03/2019 9:12 pm  

first thanks for sharing your idea

this method is just for accurate workflow or using selected strategies in two data time(e.g :2002-2012 and 2003-2013) for trading in live(e.g :2013-until tow year)?


ReplyQuote
coensio
(@coensio)
Member Admin
Joined: 5 months ago
Posts: 109
01/04/2019 9:38 am  

It is mainly meant to verify if our method can really design strategies that will be profitable in the future by 'moving' our workflow to different time points in our historical data. This eliminates 'lucky shots', or 'lucky years' where our strategies have worked only due to favorable market conditions (in IS but also in OOS periods). If you can move your workflow around to any year in the past and still be profitable in 'future' years then this gives you an additional confirmation of your strategy design and testing method.

On the otherhand, if you already have verified your workflow, then you can use the last OOS (SFT) period to see how your system would perform in this newest OOS period, which normally would be your live trading or incubating period.

 

 

 

 


sam liked
ReplyQuote
Share:
  
Working

Please Login or Register