I often hear traders dismiss entire swathes of technical analysis because they’re sick of “false signals”.

None of us like a false signal – and we like getting caught out by them even less – but deciding to ignore all signals is a bit like throwing the baby out with the bathwater.

A better solution is to get wiser and learn to recognize false signals by understanding where and why they happen.

And that’s exactly what I’d like to take a look at today.

I’m not suggesting that you’ll never get caught out again – but with practise, you can improve your chances.

Oversold markets and a false signal worry…

There’s been a lot of talk about “oversold” markets recently, but judging this kind of market strength is fraught with false signals and mixed messages.

One thing is certain – there’s no shortage of things to worry about. Take a look at my “list of doom” – it’s not happy reading … the Japanese disaster … Libyan civil war … Bahraini protests … Egyptian uncertainty … European debt problems … growth slow-down in China … unemployment … inflation …

It feels like we’re staring straight into the abyss.

Of course, the terrible events with Japan’s earthquake recently have taken centre stage. Yet, despite all this, we’ve seen strong bounces in many markets this week.

Bounces can seem to defy all logic. When the crisis deepened in Japan, amid concerns about the Fukushima nuclear plant and reports of fresh radioactive leaks the Nikkei closed up nearly 6 per cent on the day.

Market commentators like to call this “the dead cat bounce” – meaning that there is no real revival of confidence, simply a correction following a steep fall – in the case of Japan, a fall of 12 per cent since last Friday.

When a market the size of Japan loses around a fifth of its value in such a short period of time, a bounce is inevitable, no matter how uncertain the safety of the nuclear plant.

These “bounces” occur when the market is significantly oversold, so today I’d like to show you two ways to identify an oversold market – and I’d like to examine how these methods are often abused by traders, which can leave them seriously out of pocket.

RSI: a value judgement

I’ll start off with the Relative Strength Index (RSI). This is calculated based on averages of up closes and down closes over a 14-day period. It’s what’s called an oscillator – it moves between 0 and 100, with readings over 70 per cent indicating overbought, and below 30 per cent indicating oversold.

Take a look at the RSI indicator at the bottom of this chart. Every time it moves into “oversold” or “overbought” territory, we can see a correction happening in the price chart above.

Now let’s take a look at how the RSI behaved on the Japanese stock market this week.

On Tuesday, the Nikkei struck a very oversold level of 17 (see chart below), before rising back up to 30 on Wednesday. This low has only been hit on eight occasions since 1970, and each time the market’s average return after one week of doing so was over 6 per cent.

These bounces show just how willing traders are to find value wherever they believe that an instrument is undervalued.

However, RSI is far from a holy grail of “buy” and “sell” signals – if you set out to sell with the RSI is over 70 and buy when it’s below 30 – your trading fund won’t last long. Yes, it will tell us that the market is ripe for a top or bottom, but it doesn’t, in itself indicate a top or a bottom.

Stochastics: secondary signals

The second overbought/oversold indicator I’d like to take a look at is the Stochastic indicator. The Stochastic oscillator is considered a little more sophisticated that the RSI. It compares current closing prices with the recent range of high to low prices.

The premise behind it is that periods of price increases tend to show closing prices accumulating near the extreme highs of the day. And, correspondingly, during periods of price decreases, daily closes tend to accumulate near the extreme lows of the day.

Again, it’s an oscillator, so it moves between 0 and 100. Readings over 80 per cent imply overbought; below 20 per cent imply oversold.

Being in the “oversold” or “overbought” area alone is not usually regarded as a signal – but it gives the trader extra information about the state of that market. A trader acting on signals from the Stochastic oscillator will often wait for the indicator to move above the 20 per cent line or below the 80 per cent line.

The Stochastics indicator is usually represented by two lines – the first is the raw measurement, and the second is the 3-period moving average of the first – this is a smoothed out version, and is the one that most traders are interested in.

Where the line moves below 20 per cent, the instrument is considered oversold, and above 80 per cent, it’s considered overbought. As you can see from the example above, it doesn’t always work out (I’ve marked the false signals in grey).

Again, much like the RSI indicator, Stochastics aren’t reliable enough to use alone. Both throw up too many false signals and are better used as “secondary” trading tools – ones that back up your signals, giving more weight to your trades.

Learning to recognise false signals

A lot of traders overuse the terms “overbought” and “oversold” as if these will give them instant insight into market reversals.

As I’ve said, these oscillators create a significant number of false signals, and are better used for gauging market mood than actual trading signals.

Here are two situations where oscillators fail again and again – so watch out for them …

1. Oscillators are particularly unreliable in strongly trending markets – showing an overbought or oversold signal while the market continues to trend strongly.

2. RSI and Stochastics work better in a sideways market – but, watch out if the market is trading into the upper boundary of a congestion area and then breaks out on the upside. Chances are, your RSI and Stochastic indicators will tell you the market is overbought, when in reality the market is just beginning to show its upside strength.

Here’s an example where both Stochastics and RSI (many traders will wait until both are confirming a signal) are telling us to sell while the market charges upwards.

Having an understanding of how and why these false signals develop is half the battle – it makes us smarter investors, better able to judge the strength of a signal rather than simply follow it blindly.