In this chapter, you will learn to identify and describe the limitations of a study. You will learn to: Show
 The type of study and how that study is designed can determine how the results of the study should be interpreted. Ideally, a study would be perfectly externally and internally valid, but in practice this is very difficult to achieve. Practically every study has limitations. The results of a study should be interpreted in light of these limitations. Limitations can be discussed through three components:
All these issues should be considered when considering the study limitations. Almost every study has limitations. Identifying these limitation, and discussing the impact they have on the interpretation of the study results, is important and ethical. Example 9.1 (Interpretation) Smoking was once considered healthy and beneficial: some advertisements used doctors to promote cigarette smoking, and others promoted the benefits of smoking to athletes (Ingalls 1936). An advertisement for Camel cigarettes claimed that 'More doctors smoke Camels than any other cigarette' based on a survey of 113,597 doctors. That certainly is a large sample... but understanding how the data were collected is important. In fact, the company that owned Camel cigarettes conducted the survey (which raises suspicions immediately). Even worse, the company staff interviewed doctors and asked about their smoking habits after RJ Reynolds provided free cartons of Camel cigarettes to the doctors. No wonder more doctors smoked Camel brand! Concluding that 'More doctors smoke Camels than any other cigarette' in light of this may be technically correct, but is highly unethical. External validity refers to the ability to generalise the results to other groups in the population apart from the sample studied (Sect. 3.7). For a study to be externally valid, it must first be internally valid. External validity refers to how well the sample is likely to represent the target population as given in the RQ. For example, suppose the RQ is 'Among Queenslanders, what proportion own a smart speaker?'. The study is externally valid if the sample is representative of Queenslanders, and hence the results from the sample are likely to apply to Queenslanders as a whole. The results do not have to apply to people in the rest of Australia (though this can be commented on, too). The intended population, as given in the RQ, is Queenslanders. External validity depends on how the sample was obtained. Results from random samples are likely to generalise to the population and be externally valid when appropriately analysed. (The analyses in this book assume all samples are simple random samples.) Furthermore, results from approximately representative samples may generalise to the population and be externally valid if those in the study are not obviously different than those not in the study. Example 9.2 (External validity) A New Zealand study (Gammon et al. 2012) identified (for well-documented reasons) a particular group to study: 'women of South Asian origin living in New Zealand' (p. 21). The women in the sample came from a study
The results may not generalise to the intended population of 'women of South Asian origin living in New Zealand' because all the women in the sample came from only one city (Auckland), and the sample was not a random sample. The results will not generalise to all New Zealand women, but this is not a limitation: the target population was only 'women of South Asian origin living in New Zealand'. The researchers did not intend the results to apply to all New Zealand women. Example 9.3 (Using biochar) A study of growing ginger using biochar (Farrar et al. 2018) used one farm at Mt Mellum, Australia. While the results may only generalise to growing ginger at Mt Mellum, the encouraging results suggest that a wider, more general, study of the impact of using biochar to grow ginger would be worthwhile. In addition, ginger is usually grown is similar types of climates and soils, so the results may apply to other ginger farms also. Internal validity refers to how reasonable and logical the results from the study are: the strength of the inferences that can be made from the sample (Sect. 3.7). An internally valid study is effective in demonstrating that the conclusions made from the sample cannot be explained any other way. Internal validity can be compromised by confounding, the carryover effect, the Hawthorne effect, observer bias, and/or the placebo effect. Consequently, if any of these issues are likely to compromise internal validity, the implications on the interpretation of the results should be discussed. For example, if the study design implies that the Hawthorne effect may be an issue (since the participants were not blinded), this should be clearly stated, and the conclusion should indicate that the individuals in the study may have behaved differently than usual because (for example) they knew the were in a study. The internal validity of observational studies is often compromised because confounding can be less effectively managed than for experimental studies. The internal validity of experimental studies involving people is often compromised because people must be informed that they are participating in a study. Example 9.4 (Internal validity) In a study of the hand-hygiene practices of paramedics (N. Barr et al. 2017), self-reported hand-hygiene practices were very different than what was reported by peers:
When participants knew they were being studied, their responses made their own behaviours appear better than their colleagues. This is a study limitation that was necessary to discuss. Example 9.5 (Internal validity) A study of using 'vibration training' on elderly men listed some limitations of their study:
The practicality of the study results in the real world should also be discussed. This is called ecological validity. Definition 9.1 (Ecological validity) A study is ecologically valid if the study methods, materials and context approximate the real situation being studied. Studies don't need to be ecologically valid to be useful; much can be learnt under special conditions, as long as the potential limitations are understood when applying the results to the real world. Although ecological validity is not essential for a good study, ecological validity is useful if is possible to achieve. The ecological validity of experimental studies may be compromised because the experimental conditions are sometimes contrived. Example 9.6 (Ecological validity) Consider a study to determine what proportion of people will buy a coffee in a reuseable cup. People could be asked about their intentions. This may not be ecologically valid, as how people act may not align with what they say, especially when social pressures exist to use reusable cups. An alternative study could watch people buy coffees at various coffee shops, and record what people do in practice. This second study is more likely to be ecologically valid, as we are watching real-world behaviour. The limitations in a study need to be identified, and may be related to:
A study (Bingham et al. 2016) examined the effect of peer pressure from passengers among teenage male drivers. The aim was to
The use of a driving simulator was justified as:
Part of the Discussion section reads:
Later, the paper reports:
When interpreting the results of studies, we consider the practicality ( validity), the generalizability ( validity) and the effectiveness ( validity). Internal validity refers to issues such as and the Hawthorne effect. External validity refers to methods. Selected answers are available in Sect. D.9. Exercise 9.1 A student project at the university where I work had the RQ:
When discussing external validity, they said:
Why is the statement not relevant? Exercise 9.2 Despite their common use, no experimental scientific evidence shows that parachutes are effective (G. C. S. Smith and Pell 2003). To obtain evidence, researchers studied this scenario (Yeh et al. 2018). Part of the Abstract for the paper (slightly edited for clarity) says:
Based on this information:
Exercise 9.3 A study of how well hospital patients sleep at night (Delaney et al. 2018) set out to
In discussing the study, the researchers state:
Later, while discussing the limitations, the researchers state:
|
