Frequently we posts scientific articles to add value to comments, to ask if “it’s true”. For the ones like me and many others it is fairly easy to spot if a study is reliable or not. But how can the less science focused/experienced understand in what extents the study is to be trusted or has the adequate level of scientific evidence? The type of study is an important indication of how important and reliable those results can be. The source where they come from is another.
- Hierarchy of evidence (lower to higher) Note that it is a general guideline, not a rule to be taken as absolute:
- Case reports: are considered the lowest level of evidence but a good case report will be clear about the importance of the observation being reported. They are not based on systematic studies and the identified causality or association may have other explanations.
- Animal Research/Lab Studies: lower level of the hierarchy of evidence. Animal studies use animals to test drugs, medicines, etc., before evolving to human trials. As our physiology is different from other animals, a drug/medicine may show a totally different behavior in us than in animals.
- In vitro studies: laboratory trials in a controlled environment. The main problem associated with these is the different behavior drugs/chemicals show in vitro and in the human body.
- Case control studies: in these studies a comparison is made between patients with an existing condition and people free from that condition. These types of studies are usually less reliable, as they are not causal studies i.e. although they show a statistical relationship, they do not necessarily mean that one factor caused another factor. They are observational and retrospective.
- Cohort studies: these are a type of research used to investigate causes of disease. They establish links between risks and outcomes. Cohort studies observe groups of individuals, recording their exposure to the selected risk factors to understand/find the possible causes of disease. Cohort studies are also observational and establish association, not causality. They are useful as for ethical reasons, studies with a higher level of evidence such as randomized trials, cannot be performed in certain circumstances. Have no randomization, and bias can be imparted
- Randomized controlled clinical trials: these randomly (reducing bias) assign a certain number of participants to a group (experimental or control). Trials are carefully planned and introduce a treatment/exposure to understand how it effects real patients and allow comparing the intervention and control groups.
- Systematic Reviews: These consist in an extensive and comprehensive review of all relevant studies on a particular issue. All information is analyzed and reviewed and combined and the findings summarized. The results can be extrapolated and generalized, are more reliable and accurate and a resource based on evidences.
- Meta-analysis systematically combines relevant qualitative and quantitative data from selected studies to come to a single conclusion with a very high statistical power. They can establish statistical significance with studies that have conflicting results, and can be greatly extrapolated to the general population.
Look for research published in credible, peer-reviewed scientific journals. (JAMA, the New England Journal of Medicine, etc).
Personal bias is extremely important. The source of funding may completely bias the way the results are reported. E.g. in a study about the association between sugar and obesity, if the organization financing the study is a Nestlé, how reliable to you think it is? Or a study analyzing the effect of aspirin on a certain condition, financed by Bayer; Look for the funding source (its included in the paper). Bottom line: the study should be funded by someone impartial with no interest in the results.
- Sample size
The size of the sample is directly related with the amount of information collected and takes a part in determining how precise the results/estimates are. A solid, reliable and precise conclusion can only be taken with an appropriate sample size. A study that has a sample size which is too small may produce inconclusive results. N=1 experiments are hardly reliable (I do on myself as a way of understanding my own reactions to foods, etc, but that is a whole different story)