Critical Appraisal - How to evaluate a medical research article

Key message: An evaluation is always done by comparison with a standard. The standard for a research article is the logic of the research process.

 

Science has to be reproducible

Science is new, reproducible and useful knowledge. When scientist plublish new knowledge they must describe with sufficient accuracy how they gained this new knowledge. The relevant information must be presented clearly and logically to the readership. Usually this is done in a research article published in a scientific journal.

 

An organization that evaluates the quality of medical research is Cochrane:

 

"Cochrane Reviews base their findings on the results of studies that meet certain quality criteria, since the most reliable studies will provide the best evidence for making decisions about health care." (www.cochranelibrary.com/about/about-cochrane-reviews , 11.05.21)

 

Evaluating a study means to determine the extent to which the results of the study are valid. This is very hard to do when the evaluator was not involved in the study. Therefore, it is more reasonable to consider whether the results are at risk of bias than to say with certainty that they are biased.

 

Bias is not a random but a systematic error, meaning that repeating the studies in the same way will produce the same wrong results. The validity of a research study includes two aspects: internal and external validity. Internal validity is defined as the extent to which the observed results are valid in the population under study and thus are not due to systematic errors. The external validity of a study is the extent to which the results of a study can be transferred to other populations and settings.

 

"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. (www.bandolier.org/painres/download/whatis/What_is_critical_appraisal.pdf, 27.05.21)

 

Is it possible to examine trustworthiness without knowing the original data and how they were obtained? The evaluation of research is a task for professionals (Chochrane uses a handbook for their reviews). But every reader of a scientific article presenting an empirical study can come to his or her own subjective opinion if he or she understands the logic of the research process.

 

To improve the reporting of empirical studies guidelines and checklists were developed.

 

"... these recommendations are not prescriptions for designing or conducting studies. Also, while clarity of reporting is a prerequisite to evaluation, the checklist is not an instrument to evaluate the quality of observational research." (www.sciencedirect.com/science/article/pii/S174391911400212X, 10.05.21)

 

"... Trial reports need be clear, complete, and transparent. Readers, peer reviewers, and editors can also use CONSORT to help them critically appraise and interpret reports of RCTs. However, CONSORT was not meant to be used as a quality assessment instrument. (www.sciencedirect.com/science/article/pii/S0895435610001034, 10.05.21)

 

The IMRAD structure (Introduction, Methods, Results and Discussion) is recommended for the reporting of an empirical study that is based on observed or measured data. The IMRAD structure follows the logic of the research process (see my model of the research process on Learn-Study-Work: What is Science? What is Research?).

 

Example "Interventional Study"

"Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report. ... Item 20. Trial limitations, addressing sources of potential bias, imprecision" (www.consort-statement.org/Media/Default/Downloads/CONSORT%202010%20Explanation%20and%20Elaboration%20Document-BMJ.pdf, 17.06.21, p. 1)

 

We want to assess the quality of a study and not just the report on the study. So we need to examine every element of the study.

 

What are the elements of a interventional study?

  1. The objective and the research design
  2. The participants
  3. The intervention
  4. The potential confounders
  5. The outcome
  6. The study investigators
  7. The report
Research design and critical appraisal of an interventional study - www.learn-study-work.org

1. The objective and the research design

The introduction explains the rational for the study. It explains what is already known about the topic, where a knowledge gap exists, and why this knowledge gap is a problem. The objective of the study is to fill this knowledge gap and thus to help find the solution to the problem.

 

It makes no sense to set an objective without knowing how it can be achieved. Therefore, the authors must indicate the research design that they used for the study (in the titel, the abstract, the introduction or the method section). The reader should now judge:

  • Is the background knowledge adequately described?
  • Is the problem clearly explained?
  • Is the objective set reasonably?
  • Does the research design correspond to the intended objective?

2. The participants

 

 

It is often difficult to find the appropiate participants for a study.

 

"Recruitment ... begins with the identification, targeting and enlistment of participants (volunteer patients or controls) for a research study. It involves providing information to the potential participants and generating their
interest in the proposed study. There are two main goals of recruitment:
• to recruit a sample that adequately represents the target population;
• to recruit sufficient participants to meet the sample size and power requirements of the study (Hulley et al, 2001; Keith, 2001). (Patel MX, Doku V & Tennakoon T (2003) Challenges in recruitment of research participants. Advances in Psychiatric Treatment 9, p. 229)

 

The reader of a study report should judge:

  • Is the sample population appropiate to the hypothesis being tested so that the results of the study are generalisable to the target population?
    Selection bias comes about when specific kinds of people are more likely to be recruited into studies than others.
  • Is the allocation of the participants to the intervention and control group randomizes, concealed and blinded? Random allocation is a technique that chooses individuals for treatment groups and control groups entirely by chance with no regard to the will of researchers or patients’ condition and preference. Allocation concealment prevents researchers from influencing which participants are assigned to a given intervention group. Blinding refers to keeping trial participants, health-care providers, assessors or data collectors unaware of the assigned intervention, so that they will not be influenced by that knowledge.
  • Do the intervention and control group have the same baseline data?
  • Is there an attrition bias? Attrition occurs when participants leave during a study. Different rates of loss may change the characteristics of the groups.

3. The intervention