The Relevancy Trap
As traders, we can sometimes fall into the relevancy trap, focusing on and trading only those strategies developed by our own hands. We want to matter. We want to be relevant. We want our efforts to be rewarded. The best way to do that is to trade our own strategy. However, for most traders who inadvertently fall into the twin traps of curve fitting and data mining, their best efforts fall far short of what is necessary to succeed in real markets, with real money. To their surprise and dismay the Market’s Maximum Adversity starts hammering their trades, account balance, confidence and equity curve. It begins almost immediately they start trading live with real money. What they thought was a positive expectancy methodology turns into being a negative expectancy strategy, lifting their ROR above 0% and pushing their account balance towards ruin. It’s an unmitigated disappointment.

The Importance of Benchmarking
This is why it’s always important to benchmark your development efforts against established and robust methodologies. Methodologies that have worked well in the past and in all likelihood will work in the future. The benchmark strategy should be the minimum hurdle your own strategy must attain before consideration is given to trading it.

If your efforts cannot surpass the benchmark then your consideration should be given to trading it and not your own designed strategy. Do not believe yourself self-esteem, your relevancy, is attached to your development efforts. Your self-esteem and relevancy should be tied to your account balance. The objective in trading is to avoid ROR, survive, make money and build your trading account. Not pander to your narcissistic need to have your development efforts admired.

My Benchmark: The Turtle Trading Strategy
In my opinion, one of best publicly available strategies is the Turtle trading strategy.

If you’re not aware of this strategy I can recommend you read Curtis Faith’s book Way of the Turtle (McGraw-Hill 2007). Curtis Faith was one of the original Turtles taught by Richard Dennis and William Eckhardt in 1983.

The Turtle strategy is actually an adaption and improvement on Richard Donchian’s 4 Week Rule which he publically shared in the mid-1960s.

Robustness – The Number One Attribute of a Successful Strategy
The reason why I believe the Turtle strategy is so good is because it delivers in spades the number one attribute of a successful strategy, robustness.

Robustness is simply evidence or proof that a strategy has worked in the past.

Robustness is defined by a strategy’s out-of-sample performance.

The Turtle strategy has over 35 years of positive out-of-sample performance.

If you look back further to the core idea behind the Turtle strategy, Richard Donchian’s 4 Week Rule, you actually have over 50 years of positive out-of-sample performance!

Not many publically available strategies can offer the same level of positive out-of-sample performance as the Turtle strategy and that is why its one of the best to benchmark your own developments efforts against.

Robustness Ranks Ahead of Performance Metrics
As traders we can sometimes get carried away with examining a myriad of risk adjusted performance metrics like SQN, Sharp Ratio, MAR Ratio etc.  Above the line they do have relevance in helping to identify good strategies. However they’re all academic if a strategy is not robust enough to enjoy a positive out-of-sample upward sloping equity curve.

No amount of marvelous metrics will lift a falling equity curve.

In my opinion robustness, or positive out-of-sample performance, trumps the majority of performance metrics.

If a sample of proven robust strategies existed then an array of performance metrics would be useful in ranking the methodologies.

Unfortunately most strategies, below the line, are simply not robust and have little or next to no positive out-of-sample performance. Excessive curve fitting and data mining sees enthusiastic ideas crumble under the market’s onslaught. No amount of above the line positive performance metrics will help a south bound equity curve.

Robustness Builds Confidence
What is gold in strategy selection is time since release date, not perfect performance metrics. And the more time the better.

The more out-of-sample results there is the more robust a strategy is and the more confidence a trader has. Because confidence is everything when you’re in a long, deep, dark drawdown. Confidence to believe your strategy will recover and return to a new equity high. The confidence to stick to your strategy’s trade plan comes from its out-of-sample performance, not from any above the line performance metrics. The more out-of-sample performance there is the more confidence it gives us the trader. The more confidence we have in our strategy the more likelihood there is we’ll continue to trade our strategy out of its drawdown.

Certainly review and give consideration to performance metrics, however understand they’re all superfluous if your strategy is not robust to begin with.

A tradable robustness should be the Holy Grail objective for every strategy, not picture perfect performance metrics.

So as traders our singularly most importance objective is to develop a robust strategy. One that will survive the reality of volatile and unpredictable markets and deliver positive out-of-sample performance.

For me as a trader, its all about what is below the line first, and what is above is secondary.

Robustness Comes from Sound Development Principles
In a nut shell, robust strategies are those that have avoided the twin evils of curve fitting and data mining. One is always present while the other can be eliminated. Curve fitting will always be present. All traders do it to some degree. Most traders over do it. The good traders always look to minimise it. Data mining can be avoided by objectively selecting a universal portfolio of markets that are well diversified and have minimal correlation.

For my trading and research I use a universal portfolio of 24 diverse markets spread across 8 market segments that include the currencies, interest rates, indices, energy, metals, grains, softs and meats. Within each market segment I select the 3 most liquid futures contracts based on their average daily volume. Using diversification and volume as the selection criteria provides me with an objectively and independently selected portfolio of diversified markets.

A portfolio of markets that is absent of any data mining.

Benchmarking should comprise two parts;

  •     Robustness Analysis and
  •     Performance Analysis

Robustness Analysis
Unfortunately there is no single magic metric that can rank one strategy above another. I look at a collection of robustness attributes and performance measures to help me weigh-up my view on a strategy. Between the two I do give more weight to my robustness metrics, however I do attempt to balance the trade off with my preferred performance metrics.

First thing I look at is a strategy’s robustness.

Straight up I want to see what proof there is of a strategy’s robustness, positive out-of-sample performance. The more time, the more out-of-sample performance there is the more confidence I’ll have in the strategy. If there is little proof of robustness I’ll then look for positive robustness attributes. Has the strategy avoided excessive curve fitting? Is it complex or simple? Are there many or few rules? Are there many or few indicators? Are there many or few variables? Do the variables have the same value for all markets and for both buy and sell setups? Is the strategy profitable over a diverse portfolio of markets?

Once I’ve completed my robustness analysis I’ll review a number of performance measures.

Performance Analysis
I have a number of preferred performance metrics I like to review when analysing a strategy.

My number one objective in trading is to survive. Nothing comes a close second. So right up I want to know what my strategies risk-of-ruin is. Unless its 0% I’m not interested in pursuing it.

My next concern is to know what my strategy’s risk/return is. How much return has my strategy produced for its worst’s historical drawdown? It’s the old fashion corner stone of economics. Why consider a strategy if in the past it has not produced a good enough return for the amount of risk it has incurred? A simple metric? Yes. But also one that goes to the root of what we do as traders, measuring the possible return against the probable risk.

While the reward/risk ratio is good it only focuses on a single point in time when the worst historical drawdown occurred.  The Ulcer Performance Index, although more complex to calculate, gives me more granularity to the reward/risk payoff. By measuring drawdowns it delivers a more accurate risk-adjusted return picture telling me how efficient a strategy is in producing each unit of excess return per unit of drawdown risk. The higher the better.

Next I’m supper keen to know what a strategy’s average risk per trade is. I do this as I want to avoid the risk of big stops. Which leads me straight into the importance of my next measure, the strategy’s efficiency.

Efficiency with Money Management
If we survive in trading our second objective is to make money. And since we know the secret behind earning big money is money management we as traders want to know how efficient a strategy is when money management is applied.

This is terribly important because looking at a strategy’s single contract results can hide the existence of big stops.

Many strategies only look good because of their use, where the developer uses big stops to avoid being stopped out by taking profits quickly, or only when a profitable close occurs, regardless of how many days, weeks or months have gone by!

By examining the efficiency of a strategy, by reviewing their profitability with money management applied, the existence of big stops can no longer be hidden and the real power of a strategy (or lack of power) is revealed.

Difficultly in Trading
Lastly I want a quick understanding of how difficult a strategy would be to trade? How large was the worst historical drawdown? How long? How many consecutive losses has there been in the past. How smooth or dumpy will it be to trade? My preference is a strategy that is easier to trade then harder!

Strategy Review – Hard Science or Art?
Following a robustness and performance analysis you will need to tie them together. How you weigh up each is your business.

For myself I definitely give more weight to a strategy’s robustness.

The more robust a strategy is the more believable the performance measures will be. The more robust a strategy the more confidence I’ll have in trading it during a drawdown.

If I feel a strategy with little out-of-sample performance, through its simplicity, appears to be a robust strategy, I’d allow a few of the performance measures to be less then desirable.

If I feel a strategy through its complexity has been overly curve fitted I’d put it aside for no further consideration. However if I didn’t, if I felt it was so bright and shiny and  and so irresistible that I kept it on my desk for continuing review I’d expect the performance metrics to be compelling.

Whichever way you review a strategy there will be a balancing between robustness and performance measures, where for myself I give more weight to robustness attributes.

As I’ve said there is no single magic metric that ranks all strategies. Yes there is some hard science with performance measures, however there is no hard metric to weigh the robustness of a strategy.

Certainly positive out-of-sample performance provides hard evidence of robustness.

Certainly profitability over a diverse portfolio of markets demonstrates the absence of data mining.

However there certainly is no hard evidence that can determine the level of curve fitting present. Determining both the amount of, and the impact of curve fitting, is subjective. Is it excessive or is it reasonable? This is where the trader will have to rely on their experience and move more into the realm of “art” then science.

My Benchmark: The Turtle Trading Strategy
Although I regard the Turtle strategy as being the best publicly available strategy it doesn’t mean it’s either perfect or the best.

I don’t think there is such a thing as a perfect strategy and I certainly don’t think it’s the best strategy. I feel my strategies are superior.

However, for the majority of traders the Turtle strategy will be the best they have, and it should be the benchmark against which they should weigh their own efforts.

The Turtles had two variations on their strategy. For my purposes I want to focus on the initial and simplest of the two where the Turtles would enter on a 4 week breakout and place a stop at an opposite 2 week breakout.

The Turtle Trading Strategy
For Longs;
                 Entry:                A breakout of the highest high of the last 4 weeks.
                 Stop/Exit:          A breakout of the lowest low of the last 2 weeks.

For Shorts;
                 Entry:                A breakout of the lowest low of the last 4 weeks.
                 Stop/Exit:          A breakout of the highest high of the last 2 weeks.

Profit and Loss Filter:        Only trade a current signal if the previous signal was a loss.

That’s it. Simple as. And as simple does, it’s robust as.

Is it perfect? No.

But it has far more attractive attributes then the majority of self made trading strategies.

This is the strategy yours has to beat.

Let’s have a look at its robustness.

Straight off the bat its out-of-sample performance since 1983 ranks it as one of the top publically known strategies for me.

Being pattern based it has minimal curve fitting where it’s not reliant on variable dependent indicators. It has only 2 variables which are the same value for all markets and across all buy and sell set-ups. There is no massaging/curve fitting of variables to suit individual markets. They are one value for all markets across both buy and sell set-ups.

There is no data mining to boost performance as it’s profitable over a diversified portfolio of markets. There is no cherry picking.

On a robustness scale, including the Donchian 4 Week Rule, the Turtle strategy sits within a very exclusive “simplicity” club. There aren’t many more publically known strategies that are as simple and hence robust.

For an analysis of its performance metrics please refer to the individual models that are benchmarked against it.