On numerous occasions I’ve been asked the following question;
“ …Can you please tell me the SQN score of each of your strategies?…”
Well I always answer “yes” and produce the following;
SQN
- Key Level 5.1
- Key Breakout 7.5
- Key Swing 5.1
- Key Exhaustion 3.0
- Universal Trader 10.3
However, I always do it reluctantly and I always do it with the following caution.
“ … now before you get too excited I think I need to share with you my thought’s on Van Tharp’s SQN metric…”
This is where I share my thoughts. :o)
I imagine not everyone reading this will be familiar with Van’s SQN tool so I think I’ll need to spend some time unpacking it.
SQN refers to “System Quality Number”.
SQN has been designed to assist traders in determining the strengths, desirability, quality etc of a trading system. The idea behind SQN is to use it to find good quality strategies. And a good quality strategy is seen as one that is both tradable (is it easy, or is it difficult to trade?) and efficient (does it make good money when a money management strategy is applied?).
The idea is that the higher the number, the more quality, or more desirable, a strategy is.
The following values suggest the following “qualities”;
Now according to Van’s SQN tool my strategies, particularly Universal Trader, when it’s combined into a portfolio of strategies, is possibly a “Holy Grail” strategy. Now despite how good I personally think my strategies are, and they are good as I trade them each and every day, I wouldn’t place them into the “Holy Grail” category.
As I say to people, they shouldn’t get too excited by my strategy’s SQN scores.
So please let me share my thought’s on Van’s SQN metric.
First up, for a detailed explanation traders need to refer to his book Definitive Guide to Position Sizing (The Van Tharp Institute, 2013).
However for my purposes here I’ll try to give you a thumb nail overview.
First up I want to applaud Van for his ambition to develop a single metric to help traders distinguish between poor and good quality strategies.
There is no doubt about what his System Quality Number is all about. Does a strategy you’re looking at score low or high? Is it a low or high quality strategy?
Attempting to distill the decision between a low or high quality strategy down to a single metric should be applauded as we as traders spend so much time and effort in investigating, developing, reviewing and throwing out strategies. To have a single metric to give either a thumbs up or thumbs down decision, would not only be a useful tool, but it would save so much time, effort, resources and sanity!
So a big thumbs up here for Van’s ambition in developing his System Quality Number tool.
However, unfortunately for me, and I’m only talking for myself here, I haven’t been, and I’m still not convinced, it achieves its objective in being a reliable or valuable metric in helping traders distinguish between low quality and high quality strategies, or being bad or good strategies.
Look, I know it would be easy for me to say “… here are the SQN score for my strategies..” and leave it at that, knowing you’d be impressed. However that would be disingenuous of me knowing what I know. So I can’t just give you the scores and let you think you may be looking at the “Holy Grail”!
So let’s take a reasonable dive into what the SQN is and see whether I can explain my reservations.
Here’s the formula;
As you can see the SQN is measuring the relationship between a strategy’s average reward to risk ratio and the standard deviation of its distribution and the number of trades it makes.
As a side bar Van refers to the reward/risk ratio as the profit/risk ratio or “R”, or “R-Multiple”, where “R” refers to risk.
I personally prefer to view it as the old fashion reward to risk ratio.
As I’ve said the idea behind SQN is to use it to find good quality strategies. And a good quality strategy is seen as one that is both tradable (is it easy, or is it difficult to trade?) and efficient (does it make good money when a money management strategy is applied?).
According to the above formula, for the SQN to record a high value (and hence indicate a high quality strategy) it requires one or two things to occur.
Firstly, it will either need a relatively smaller or narrower distribution of its individual trades’ reward/risk ratios to its average reward/risk ratio, or a lower standard deviation, and/or.
Secondly, it will need a high number of trades.
Achieve either one, a narrower distribution in the reward/risk ratios, or a higher number of trades, and you’ll record a higher SQN score, moving it away from being considered poor to being seen as good.
The formula’s simple maths clearly shows where the value drivers for “quality” come from and on an initial instinctive level, it makes perfect sense.
Firstly, a lower standard deviation suggests the majority of trade results will cluster around its average making it much easier to trade compared to say a strategy with a relatively higher standard deviation where trade results do not cluster around its average. An easier to trade strategy, on an instinctive level, would indicate a higher quality strategy.
Secondly, a strategy that produces more opportunities to trade means it should be making more money and hence instinctively would indicate a higher quality strategy.
So a strategy scoring a high SQN value would seem to suggest it’s either easier to trade or it makes more money because it generates more trades. Either one suggests, according to SQN, it’s a high quality strategy.
Now that we have a better understanding of the mathematical construction of SQN let me now discuss my reservations.
In a nut shell it seems to me its arbitrary to suggest “easier” to trade and “more” trades should be the final arbiter of “quality”? I think to anoint “easier” to trade and “more” trades as the gold standard for strategy quality is incorrect and possibly too biased and too subjective to be relied upon.
This brings me to my first concern about Van’s SQN.
First Concern: SQN is Biased
Let’s look at the first part of its calculation;
As I’ve mentioned SQN will give a higher score to strategies with a relatively lower standard deviation of its reward/risk ratio distributions.
So straight up it defines a “quality” strategy as being one that has a relatively narrow distribution of reward/risk ratios. It has basically picked a winner before the race has even started. In its eyes, narrower distribution strategies have more quality, and hence are superior, to wider distribution strategies.
I think defining “quality” based on the distribution is to forejudge and is prejudicial, and I believe, a bridge too far.
To my eyes SQN has a built-in bias towards mean-reverting strategies. It favours them over trend-trading strategies.
Mean-reverting strategies tend to have lower standard deviations with their many small wins and few large losses when compared to trend trading strategies with their many small losses and a few huge wins. Trend trading strategies have a much wider distribution of reward/risk ratios. SQN’s construction sees it being prejudice against trend trading while being overly biased towards mean-reversion trading.
It’s already preordained what “quality” is before making any calculations!
Unfortunately, in my eyes, SQN is not impartial in its judgements.
Yes, a higher SQN number would suggest its easier to trade, and yes, mean-reversion strategies are easier to trade compared to trend-trading, but that doesn’t necessarily make them better or higher “quality” strategies.
The second part of the calculation shows another bias.
It clearly shows that strategies with more trades will score a higher SQN value. Once again its showing a biasness towards higher frequency strategies such as mean-reversion methodologies.
So this is another issue I have as I can’t accept that more trading necessarily implies more “quality”, but SQN does.
So on two fronts, SQN exhibits a clear bias towards mean-reversion strategies and in my opinion incorrectly preordains narrow distributions and higher trade frequencies as arbiters of quality.
Though I can accept “quality” can be a tangible objective while its identification can be a little subjective (as we all know beauty is in the eye of the beholder), I feel SQNs clear biasness towards narrow distributions and higher frequency trading makes it too conflicted to be accepted as a universally impartial and objective measurement of “quality”.
Second Concern: SQN is Not Consistent/There’s too Much Variability
Now Van does acknowledge the huge impact large numbers do have on SQN’s calculation. As a work around he suggests traders use “N=100” for when there are more then 100 trades. Let’s have a look.
Well, that went south quickly. Universal Trader went from being a possible “Holy Grail” strategy to one that is “very hard to trade”.
Wow, what a reversal in fortune.
This change brings three issues to hand for me.
Firstly, if SQN is looking at how efficient a strategy is in making money, achieving our objective as traders, then I’d think using the total trade sample size is important, as opposed to an arbitrary use of “N=100”, because the more trades you have the more opportunity you have to earn expectancy and hence money. I’m not advocating that more trades means more quality, no. I’m just saying if SQN wants to identify good quality strategies that are efficient in making money, then the formula should use the total number of trades as it’s so important in determining efficiency in making money (please refer to my note at the end regarding CAGRs).
The second issue is the variance in SQN scores. Using all trades Universal Trader scores 10.3, using “N=100”, it only manages to register 1.0. That’s a huge difference and represents so many degrees of variance that it really brings into question the validity of SQN. There is so much flex in the score it really questions its credibility, well certainly in my eyes it does.
And finally, it’s not consistent in its ranking. When using all trades Universal Trader scores the highest SQN value, as it should since it represents the sum of its individual parts. However, when Van’s “N=100” work around is used, Key Swing records the highest SQN value. This inconsistency brings into question the robustness of SQN as a useful tool.
Third Concern: SQN fails to Correctly Identify Quality
I applaud Van Tharp’s ambition to develop an insightful tool, however I feel SQN fails to define and measure a strategy’s “quality”.
It’s built-in biases towards narrow distributions and high trade mean-reversion strategies does not make it impartial between strategies, and its anointment of narrow distribution and high trade frequency as being attributes of quality is subjective, narrow and misleading.
Although I’d be the first to use a System Quality Number tool, if it was impartial, objective and accurate, however because it isn’t, I can’t.
Let me add two strategies to mine and review how SQN treats them.
Let’s take a look.
DCB refers to Richard Donchian‘s 4 Week Rule strategy or “Donchian Channel Breakout” and “Turtle” refers to the popular Turtle 4 week channel breakout strategy that was built upon Donchian’s 4 Week Rule.
Quite simply I believe both are “quality” strategies and I say this for one reason.
I know “quality” can be subjective however I feel there is one immutable attribute a “quality” strategy should have, and that is robustness. A positive out-of-sample performance since release. Quite simply, does a strategy have “time” since release to demonstrate it works? Can it provide any objective evidence it has an edge like the DCB when it was first shared in the 1960s and the Turtle strategy when it was taught in 1983? That is 60+ and 37+ years of positive out-of-sample performance, or “quality” to the power of infinity.
For me, that is the number one attribute I want in a strategy I trade. Not a high score of a perfect metric. No amount of marvelous metrics will tell me a strategy is of a certain quality unless there is evidence of robustness. Evidence of “time” since release.
According to SQN, where Van suggests using “N=100” when there are over 100 trades, both the DCB and Turtle strategies score under 1. According to Van it implies the strategies are hard to trade, which is true as they are trend following, however it also implies they are low quality strategies. Which they certainly are not. However according to SQN, they are.
According to SQN, when we use all the trades, SQN scores Turtles at 5.8, indicating it’s a superb strategy. However “superb” implies it’s easy to trade, when it’s not.
This is another example of SQN not being consistent, and generating too much variability.
In my opinion I believe it’s incorrect to label the popular Turtles strategy as a low quality strategy when it may be, or close variations of it, are possibly responsible for more funds under management being traded by CTAs then any other strategy. A low quality strategy? No. I’m sorry. It’s a high quality strategy. But not according to SQN.
As I’ve said, traders shouldn’t get too excited by my strategy’s SQN scores.
Is there an Alternative to the SQN?
If SQN fails in its objective, is there an alternative?
I don’t believe there is a single metric that can help traders rule one strategy over another. I certainly don’t have one.
However I do acknowledge the real need for traders to have the ability to objectively and as accurately, if possible, to be able to sort the wheat from the chaff when it comes to reviewing and identifying good quality strategies.
For me personally I use a combination of robustness and performance measures.
I talk about them at length in my book The Universal Tactics of Successful Trend Trading (Wiley, 2020).
Please refer to Chapter 8.
However I can address one of SQN’s objectives which is to determine how efficient a strategy is in making money. Rather then using SQN I personally use the CAGR (Compound Average Growth Rate).
Let’s have a look.
CAGR is an easy calculation showing how efficient a strategy is in making money when a money management strategy is applied. In the example above I have started with a $50,000 account using Fix Percentage Money Management where I have risked 2% per trade. As you can see Universal Trader is ranked the highest with a 32% CAGR, however using Van’s “N=100” work around its Key Swing which ranks higher, yet it only manages a 26% CAGR.
Now please note the CAGR only tells me how efficient a strategy is in making money. It doesn’t tell me what level of risk it took to make the money. You will also need to review risk adjusted return measures to help in your assessment of whether a strategy is of low or high quality.
From my experience you need to use a number of robustness and performance tools to assist you in determining whether one strategy is superior to another. I wish there was one magic single metric available to help us however I have yet to find one.
As I say please refer to my book The Universal Tactics of Successful Trend Trading (Wiley, 2020) where I discuss how I attempt to sort the wheat from the chaff.
Brent Penfold
Sydney