Key message: An evaluation is always done by comparison with a standard. The standard for a research article is the logic of the research process.
Science is new, reproducible and useful knowledge. When scientist plublish new knowledge they must describe with sufficient accuracy how they gained this new knowledge. The relevant
information must be presented clearly and logically to the readership. Usually this is done in a research article published in a scientific journal.
An organization that evaluates the quality of medical research is Cochrane:
"Cochrane Reviews base their findings on the results of studies that meet certain quality criteria, since the most reliable studies will provide the best evidence for making decisions about health care." (www.cochranelibrary.com/about/about-cochrane-reviews , 11.05.21)
Evaluating a study means to determine the extent to which the results of the study are valid. This is very hard to do when the evaluator was not involved in the study. Therefore, it is more reasonable to consider whether the results are at risk of bias than to say with certainty that they are biased.
Bias is not a random but a systematic error, meaning that repeating the studies in the same way will produce the same wrong results. The validity of a research study includes two aspects: internal and external validity. Internal validity is defined as the extent to which the observed results are valid in the population under study and thus are not due to systematic errors. The external validity of a study is the extent to which the results of a study can be transferred to other populations and settings.
"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. (www.bandolier.org/painres/download/whatis/What_is_critical_appraisal.pdf, 27.05.21)
Is it possible to examine trustworthiness without knowing the original data and how they were obtained? The evaluation of research is a task for professionals (Chochrane uses a handbook for their reviews). But every reader of a scientific article presenting an empirical study can come to his or her own subjective opinion if he or she understands the logic of the research process.
To improve the reporting of empirical studies guidelines and checklists were developed.
"... these recommendations are not prescriptions for designing or conducting studies. Also, while clarity of reporting is a prerequisite to evaluation, the checklist is not an instrument to evaluate the quality of observational research." (www.sciencedirect.com/science/article/pii/S174391911400212X, 10.05.21)
"... Trial reports need be clear, complete, and transparent. Readers, peer reviewers, and editors can also use CONSORT to help them critically appraise and interpret reports of RCTs. However, CONSORT was not meant to be used as a quality assessment instrument. (www.sciencedirect.com/science/article/pii/S0895435610001034, 10.05.21)
The IMRAD structure
(Introduction, Methods, Results and Discussion) is recommended for the reporting
of an empirical study that is based on observed or measured data. The IMRAD structure follows the logic of the research process (see my model of the research process on Learn-Study-Work: What is Science? What is Research?).
"Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report. ... Item 20. Trial limitations, addressing sources of potential bias, imprecision" (www.consort-statement.org/Media/Default/Downloads/CONSORT%202010%20Explanation%20and%20Elaboration%20Document-BMJ.pdf, 17.06.21, p. 1)
We want to assess the quality of a study and not just the report on the study. So we need to examine every element of the study.
What are the elements of a interventional study?
1. The objective and the research design
The introduction explains the rational for the study. It explains what is already known about the topic, where a knowledge gap exists, and why this knowledge gap is a problem. The objective of the study is to fill this knowledge gap and thus to help find the solution to the problem.
It makes no sense to set an objective without knowing how it can be achieved. Therefore, the authors must indicate the research design that they used for the study (in the titel, the abstract, the introduction or the method section). The reader should now judge:
2. The participants
It is often difficult to find the appropiate participants for a study.
"Recruitment ... begins with
the identification, targeting and enlistment of participants (volunteer patients or controls) for a research study. It involves providing information to the potential participants and generating
interest in the proposed study. There are two main goals of recruitment:
• to recruit a sample that adequately represents the target population;
• to recruit sufficient participants to meet the sample size and power requirements of the study (Hulley et al, 2001; Keith, 2001). (Patel MX, Doku V & Tennakoon T (2003) Challenges in recruitment of research participants. Advances in Psychiatric Treatment 9, p. 229)
The reader of a study report should judge:
3. The intervention