Health surveys – what do they tell us?

Guest blog by Bob Mathers, management skills trainer, writer for Hospital Doctor online and also my Dad. In this post Bob explores what health surveys actually tell us.

Health surveys have been around for a while, but what do they really tell us? We should ask a few simple questions:

• Who carried out the survey?
• Who did they carry out the survey for?
• Who is sponsoring this survey on garlic pills, sun cream or low fat diets?

More often than not it’s a company with something to sell. In which case you can bet your bottom dollar that so-called facts or findings may not be what they seem.

By printing an ‘independent’ survey, a publication is suggesting that there may be something in it, while remaining at arms’ length from any responsibility for the findings.

Could it be that one of its advertisers would benefit from any good PR from this survey?

How are samples collected?

Let’s look at the technical questions:

• How are samples selected?
• How are people approached and questioned?

This is before any data has been collected or analysed.

Sample selection

There is an immediate ’selection bias’ built in, particularly in postal surveys, where respondents are openly targeted according to income or a previous purchase made or by their address, postcodes being widely used to define the lifestyles or net worth of residents.

Most of those who respond to postal surveys do so because they have an interest in the subject. This is known as ‘ascertainment bias‘. When you read: ‘This survey shows,’ beware. What is not pointed out is that of 1,000 people questioned, only 13% replied and 87% did not.

Of the 13% who replied, more than half said one thing, while less the half said the other. By the time it is reported, ‘the survey shows that 62% of respondents said x,’ when the more truthful and mathematically correct interpretation should be, ‘only 8% of the total survey population said x’.

But who said statistics had anything to do with either truth or mathematics?

How should people react to being surveyed?

When people are questioned face to face, how do we know that they do not just give the answers they think the interviewer wants to hear, rather than forcefully express their opinion? Do we know what questions were asked, how they were framed, whether they were leading questions?

Few surveys list the questions asked in appendices. Because of their personal nature, health surveys can also reveal people’s biases or social sensitivities, particularly if the questions are poorly framed.

For example, advertising campaigns in the mid-80s associated the consumption of animal fat with coronary risk so that spreading margarine suddenly became socially acceptable and even an indication of a prestigious (because stressful) life. Yet not long before the same margarine was regarded as an indication of lower class poverty and it was difficult to get housewives to admit that they used it instead of butter.

The reason behind the new consumer attitude was the subtle changes made to the survey questions being put.

So did the consumers change from butter to margarine for health or social reasons?

We never find out.

The real driver behind any survey is cost

A sample should yield enough individuals in each critical group for us to be confident that they are truly representative – but each interview costs money.

Add to this the more comprehensive a questionnaire becomes, the more it puts off respondents, demoralises interviewers and wastes time and money.

Given this context it is worth determining what, if any, value health surveys really bring and how much they are pawns in the publication price or brand wars. It could be argued that they tend to have a health bias – ‘Joining the local health/sports/fitness club can improve your performance’.

The implication here is that x% of people surveyed felt fitter or had more energy because they joined a club. But did it also capture the opinions of those who did not join?

• How many did not join for economic reasons but worked out on their own, at no cost?
• Were they less fit?
• Was there a comparison between the groups?

We never find out.

More people fear public speaking than death

In a 1993 study of human fears done by the polling firm Bruskin-Goldring in the United States, 30% of those surveyed said they feared death while 45% said they feared public-speaking.

It is comforting to know there are worse things than death.

Whatever health surveys you read in future, take them with a pinch of salt – just how much depends on what survey you last read.

Bob Mathers writes regularly for Hospital Doctor online on ’soft’ management skills such as communication and time management. This article first appeared in 2001 and has been re-edited for this guest blog post. If you enjoy it, why not read his other guest post in this blog: No Bad Thing on how the English language changes.