257
Speaking of research
Guidelines for evaluating research articles
Phillip Rumrill∗, Shawn Fitzgerald and
Megen Ware
Kent State University, Department of Educational
Foundations and Special Services Center for
Disability Studies, 405 White Hall, P.O. Box 5190,
Kent, OH 44242-0001, USA
The article describes the components and composition of
journal articles that report empirical research findings in the
field of rehabilitation. The authors delineate technical writing
strategies and discuss the contents of research manuscripts,
including the Title, Abstract, Introduction, Method, Results,
Discussion, and References. The article concludes with a
scale that practitioners, manuscript reviewers, educators, and
students can use in critically analyzing the content and scientific merits of published rehabilitation research.
Keywords: Assessment, research articles, guidelines for critique
1. Introduction
The purpose of this article is to examine the components of a research article and provide guidelines
for conducting critical analyses of published works.
Distilled from the American Psychological Association’s [1] Publication Manual and related descriptions
in several research design texts [4,8,9,12,15], descriptions of how authors in rehabilitation and disability
studies address each section of a research article are
featured. The article concludes with a framework that
rehabilitation educators, graduate students, practitioners, and other Work readers can use in critiquing research articles on the basis of their scientific merits and
practical utility.
∗Corresponding author: Tel.: +1 330 672 2294; Fax: +1 330 672
2512; E-mail: prumrill@educ.kent.edu.
2. Anatomy of a research article
For nearly 50 years, the American Psychological Association has presented guidelines for authors to follow
in composing manuscripts for publication in professional journals [1]. Most journals in disability studies
and rehabilitation adhere to those style and formatting
guidelines. In the paragraphs to follow, descriptions
of each section of a standard research article are presented: Title, Abstract, Introduction, Method, Results,
Discussion, and References.
2.1. Title
As with other kinds of literature, the title of a scientific or scholarly journal article is a very important feature. At the risk of contravening the age-old adage “You
can’t judge a book by its cover,” Bellini and Rumrill [4]
speculated that most articles in rehabilitation journals
are either read or not read based upon the prospective
reader’s perusal of the title. Therefore, developing a
clear, concise title that conveys the article’s key concepts, hypotheses, methods, and variables under study
is critical for researchers wishing to share their findings
with a large, professional audience. A standard-length
title for a journal article in the social sciences is 12–15
words, including a sub-title if appropriate. Because social science and medical indexing systems rely heavily
on titles in their codification schemes to track and categorize journal articles by topic, providing a title that
clearly delineates a general research domain or topic
area is of utmost importance. If the title is vague or
ambiguous, chances are that the prospective reader will
not continue to read through the document to establish
where it might fit in terms of a specific research domain
or topic area. Examples of clearly descriptive titles
that can be found in the contemporary rehabilitation
literature include:
“Rehabilitation Counselors’ Assessments of Applicants’ Functional Limitations as Predictors of Rehabilitation Services Provided” [3].
Work 14 (2000) 257–263
ISSN 1051-9815 / $8.00  2000, IOS Press. All rights reserved
258 P. Rumrill et al. / Guidelines for evaluating research articles
“Employer Concerns About Hiring Persons with
Psychiatric Disabilities: Results of the Employer Attitude Questionnaire” [6].
“Self-Perceived Reasons for Unemployment Cited
by Persons with Spinal Cord Injury: Relationship to
Gender, Race, Age, and Level of Injury” [13].
“Vocational Rehabilitation Counselors’ Attitudes
Toward Self-Employment Outcomes” [18].
“Surveying the Employment Concerns of People
with Multiple Sclerosis: A Participatory Action Research Approach” [20].
“Effect of Graduate Research Instruction on Perceived Research Anxiety, Research Utility, and Confidence in Research Skills” [21].
Before we move into descriptions of the content sections of a research article, we want to briefly address
the concept of technical writing as it applies to the composition of academic manuscripts. Journals adhering
to the American Psychological Association’s [1] publication guidelines favor manuscripts that are written
in direct, uncomplicated sentences. Editors prefer that
text be written in the “active voice”; whenever possible,
sentences should begin with their subjects and follow
with verbs and objects (e.g., “The researcher conducted
an experiment” rather than “An experiment was conducted by the researcher”). Technical writing is marked
by the “less is more” maxim; extraneous phrases and
clauses that add words to the sentence without enhancing the overall statement should be avoided (e.g., “In
order to… ”, “For purposes of… ”, “As far as… is
concerned… ”). Another element of sound technical
writing is the sparing use of adverbs (e.g., very, somewhat, strikingly) and adjectives that do not serve to further define or specify the terms that they are modifying
(e.g., interesting, important, good, noteworthy).
In addition to the American Psychological Association’s guidelines for technical writing, authors should
consider these six criteria for effective composition
provided by George Orwell (1946) in Politics and the
English Language:
1. Never use a metaphor, simile, or other figure of
speech which you are used to seeing in print.
2. Never use a long word where a short one will do.
3. If it is possible to cut a word out, always cut it
out.
4. Never use the passive (voice) where you can use
the active.
5. Never use a foreign phrase, a scientific word, or
jargon word if you can think of an everyday English equivalent.
6. Break any of these rules sooner than say anything
outright barbarous (p. 170).
Organization is also key in preparing an effectively
composed journal manuscript, with multi-level headings serving to guide the flow of text and keep the
reader on track. For authoritative information regarding the style and formatting guidelines for submitting
manuscripts to most journals in social science fields,
readers should consult the American Psychological Association’s [1] Publication Manual. For information
concerning the style and formatting requirements of
Work and other journals published by IOS Press, see
the Guidelines for Authors section included in the beginning of this edition.
2.2. Abstract
Next to the title, the abstract is the most widely read
section of a journal article. In an empirical article, the
abstract should be a succinct, 100–150 word summary
of the investigation’s key features, including purpose,
objectives, research questions/hypotheses, sample, scientific procedures, independent and dependent variables, and salient results. Results of the study should
be summarized in full in the abstract; authors should
describe both significant and non-significant findings,
not only those which upheld their hypotheses or expectations. The abstract serves as an advance organizer
for the article, and it should include every important
premise, method, and result of the investigation. Like
the Preface that commonly orients readers to full-length
textbooks, the abstract provides a complete, albeit summary, preview of the article. Some journals, including Work and the Journal of Vocational Rehabilitation,
ask authors to list key descriptors on the abstract page,
which are then used for purposes of indexing. In most
cases, the title is what determines whether a reader will
read the abstract; the abstract determines whether the
reader will read the body of the article.
2.3. Introduction
Immediately following the abstract, the introductory
section of the article sets the stage for the study upon
which the article was based. It orients the reader to the
problem or issue being addressed, develops the logic
and rationale for conducting the investigation, and almost always expresses the empirical hypotheses or research questions. Heppner et al. [9] suggested that
the introduction should answer questions such as why
the topic is an important one to study, what previous
P. Rumrill et al. / Guidelines for evaluating research articles 259
work bears on the topic, how existing work logically
connects to the author’s research questions and/or hypotheses, how the question will be researched,and what
predictions can be made.
To answer these questions, authors typically address
three major elements in the introductory section of an
article: (1) The Research Problem, (2) The Framework
for the Study, and (3) The Research Questions and Hypotheses [8,15]. We will describe each of these introductory elements in linear fashion, but we do not mean
to imply an order in terms of how they should be addressed. Many (if not most) authors blend these considerations to fit the flow and logic of their respective
manuscripts.
The research problem. Usually in the very first sentences of an empirical journal article, the author draws
the reader’s attention to the scope, impact, and current
status of the problem or issue being investigated. This
orientation is most effectively achieved by applying the
broadest-possible perspective to the concern. A study
of success rates among participants in a stress inoculation program for people with diabetes mellitus might be
introduced by citing national statistics concerning the
incidence and prevalence of this very common disease.
An article describing the effects of a model job placement program for women with breast cancer might begin with a review of existing literature concerning employment and breast cancer, with a particular focus on
the difficulties that women have in re-entering the labor force following diagnosis and treatment. Authors
reporting a longitudinal study of the post- school employment outcomes of secondary students with developmental disabilities would likely introduce their article with a review of the disappointing adult outcomes
which that population has experienced since the inception of formalized transition services in the mid–1980s.
The framework for the study. The specific theoretical and empirical framework for the particular investigation is another important part of the Introduction.
Authors summarize existing literature related to the
identified problem, then build a logical rationale for a
study that addresses gaps or inconsistencies in the literature. The author should present the theoretical or
conceptual model that informs the inquiry and provides
enough background to enable the reader to appreciate
the rationale of the current study. This framework elucidates the purpose of the current study (e.g., to evaluate the effectiveness of a job placement program for
women with breast cancer), which is then operationalized in the research questions or hypotheses. Social
scientific theories which have figured pominently in
the frameworks of recent rehabilitation investigations
include Hershenson’s [10] model of work adjustment,
Bandura’s [2] concept of situational self-efficacy, and
Bolton and Brookings’ [5] integrated model of empowerment.
The research questions and hypotheses. The Introduction section of a research article typically includes
a statement of the research questions and/or hypotheses that served to guide the study. A more speculative research question tends to be used in descriptive
research designs (e.g., surveys, program Assessments,
empirical literature reviews) or in qualitative studies.
Examples of research questions could include: “What
concerns do college students with disabilities have regarding their future career prospects?”; “What themes
are evident in the psycholinguistic development of deaf
women?”; and “What steps are Fortune 500 employers taking to provide on-the-job accommodations for
workers with disabilities?”.
The hypothesis, on the other hand, is predictive by
design. Its specificity is dependent upon the theory underlying it or previous, relevant research, but it should
include the direction of the expected results whenever possible. Independent and dependent variables
need not be operationalized in theory-based hypotheses
(because this is done in the Method section), but the
expected relationship among study variables must be
clearly articulated. Examples of directional hypotheses
could include: “Participation in a cognitive-behavioral
stress inoculation program will decrease symptom onset and magnification”; “Anxiety, depression, and low
self-esteem will be collectively, positively, and significantly related to work interference”; and “Rehabilitation counselors will rate people with severe disabilities as less favorable candidates for employment than
similarly qualified people with mild or no disabilities”.
2.4. Method
The Method section delineates how the research
questions were addressed and/or how the hypotheses
were tested. It should provide the reader with sufficient
information so that one could replicate the investigation, and it should leave no question as to what was
“done” to the participants. Because the Method section
is the primary source for determining the validity of the
study [4], the quality and clarity of this section are generally regarded as the strongest determinants of whether
an empirically-based manuscript will be accepted for
publication [9,16].
260 P. Rumrill et al. / Guidelines for evaluating research articles
Although the type and order of sub-sections found
in the Method section of a research article vary depending upon the design of the study and the author’s
judgement related to the flow of text, most articles include descriptions of the study’s subjects/participants,
instruments/measures/variables, materials, design, and
procedures.
Subjects/participants. According to Heppner et
al. [8,9], the Method section should include (a) the total
number of subjects and numbers assigned to groups, if
applicable; (b) how subjects were selected and/or assigned; and (c) demographic and other characteristics
of the sample relevant to the study’s purpose. Some authors also include a description of the population from
which the study sample was drawn, a description of the
specific sampling procedure used (e.g., simple random,
stratified, cluster; [4]), an indication of the representativeness of the sample vis a vis the broader population, the circumstances under which subjects participated (e.g., whether they were compensated, what risks
they assumed), statistical power analyses, and response
rates (if applicable).
Instruments/measures/variables. The Method section must include a detailed description of how all study
variables were operationalized, measured, scored, and
interpreted. All instruments or measures that were used
in sampling, conducting the study, and evaluating results must be specified in terms of content (e.g., number of items, response sets), how measures were administered, scoring procedures, relationship to study
variables, and psychometric properties (e.g., standardization, reliability, validity). Authors should also include a rationale for selecting each instrument, that is,
why that instrument was the best choice for measuring
a particular construct.
Materials. Researchers should also include a description of any materials that were used to carry out
the investigation. Written guides for participants, instructional manuals, media or technology, and scientific apparatus or equipment should be described in detail. Some authors include a description of the setting
in which the study was executed or data were collected.
Design. One of the most important features of the
Method section is a clear description of the design of
the study. This is essential because the design serves as
the link between (a) the research questions/hypotheses
and the scientific procedures used in carrying out the
study and (b) the findings of the study and how these
are interpreted. Authors typically label their designs
in terms of how variables were manipulated, observed,
and analyzed. Thereby, the design is the unifying force
in connecting the research objectives to both the results
and the knowledge claim that is made. To every extent
possible, a direct reference to the hypotheses should
be made when authors identify the design of a particular investigation. For example, Rumrill, Roessler, and
Denny [19] described their design as follows: “The researchers selected a three-group, posttest-only (experimental) design to assess the intervention’s univariate
and multivariate effects on (a) self-reported attitudes
(situational self-efficacy and acceptance of disability)
and (b) participation in the accommodation request process.”
Procedures. The most important component of the
Method section is the easiest to describe. In chronological order, authors simply list every step they took
in developing, administering, and evaluating the study.
Beginning with the recruitment of participants, following the study through collection of the last datum, and
including everything in-between – the Procedures subsection should provide the reader with a step-by-step
protocol that could serve as a guide for replicating the
study. Descriptions of any interventions should be provided in detail, along with summaries of the qualifications of project personnel who were instrumental in
executing the investigation. Procedures should also include how the investigation ended, along with a statement of any debriefing or follow-up services provided
to participants.
2.5. Results
The Results section of a research article should include a complete inventory of all relevant findings obtained by the investigators. In articles that report quantitative studies, results are typically presented in two
parts – (a) summary, or descriptive, statistics related
to participants’ performance on the measures that were
taken (e.g., means, standard deviations, frequencies,
percentages) and (b) statistical analyses related to the
specific hypotheses of the study (e.g., analysis of variance, multiple regression, factor analysis). We believe
that all analyses conducted as part of the investigation
should be reported in full, not only those which yielded
statistically significant results. The Publication Manual of the American Psychological Association [1] provides considerable guidance related to how statistics
should be presented in the Results section, but it does
not always provide adequate guidelines regarding what
statistical information should be included. Heppner et
al. [9] identified a pattern in recent social science literature whereby researchers tend to err on the side of
262 P. Rumrill et al. / Guidelines for evaluating research articles
providing too little statistical information: “The trend
has been to report less; for example, one rarely sees
analysis of variance source tables anymore. More disturbing is the tendency not to report important information (such as size of test statistic and probability levels)
when results are non-significant. This minimalist point
of view puts the emphasis on statistical significance and
ignores concepts such as effect size, estimation, and
power.”
In recent years, the “minimalist” perspective (in
terms of reporting statisitical findings) has been challenged by numerous researchers and statisticians [11,
14,22]. The most serious argument against this perspective relates to the influence that sample size has
in determining the significance of any statistical test.
Hayes [7], for example, pointed out that virtually any
study can be made to yield statistically significant results if the researcher includes enough subjects. To
avoid the possibility of misleading research consumers,
the latest edition of the Publication Manual [1] suggests
that all authors provide estimates of practical or clinical
significance along with all statistical significance tests
reported in the Results section.
A quantitative Results section should be limited to
the findings obtained by the researcher(s) in the current investigation. Speculation concerning what those
findings mean in a larger context is reserved for the
Discussion section.
The Results sections of qualitatively oriented articles
display much more variety in the content and manner of
presentation than is found in quantitative studies. Because the researcher’s subjective interpretations help to
shape the processes and outcomes of qualitative investigations, results are often framed in broad, interpretive
contexts. In that regard, the lines between the Results
and Discussion sections are often blurred in qualitative
research.
Researchers (qualitative and quantitative) commonly
use tables and figures to summarize and/or graphically
present their results. There is wide variability in the
content and presentation of tables and figures, with
the most important universal requirement being easy
interpretability for the reader.
2.6. Discussion
The Discussion section serves as the researcher’s forum to go beyond the current investigation and discuss
the contributions of study findings to existing literature, theory, and professional practices. The first part
of a thoughtful Discussion is often an analysis of the
study’s results vis a vis the research questions and hypotheses. Researchers should begin with a discussion
of whether the hypotheses were upheld, posit possible
explanations for those outcomes,and draw implications
from the findings back to the research problem that was
identified in the Introduction. If the results provide
a warrant for modifying or re-testing the conceptual
framework upon which the investigation was based, the
Discussion section is the place to suggest a reformulation of the underlying theory. Researchers should also
include a statement of the scientific limitations of the
current study, along with specific recommendations for
future research. Finally, the researcher ends the article with a cogent summary of the conclusions, in the
most general sense, that can be drawn from the methods
and findings of the current study. Some authors use a
separate Conclusion section for this purpose.
2.7. References
The final section of a research article is always a
listing of the references that were cited in the body of
the text. References are listed in alphabetical order,
according to authors’ last names. Most rehabilitation
journals require adherence to the American Psychological Association’s [1] guidelines regarding the composition of the References section.
3. A scale for critiquing research manuscript and
articles
Understanding the components, organization, and
composition of a research article will help make Work
subscribers better informed consumers as they read empirically based publications. As readers digest the contents of research articles and apply them to their practices, the “anatomy” of research reports can serve as a
useful rubric for critically analyzing the quality, content, and practical significance of published articles.
Table 1 presents specific questions for conducting a
section-by-section critique of a rehabilitation research
article.
4. Conclusion
This article examined the components of a research
article and provided guidelines for conducting a critical
analysis of published research. Although the descriptions of the components of a research article provide
P. Rumrill et al. / Guidelines for evaluating research articles 263
only a skeletal summary of what should be included
in a published research article, they should provide the
reader enough information to both prepare manuscripts
for publication and evaluate the empirical research that
appears in Work and other rehabilitation journals.
References
[1] Washington, D.C., American Psychological Association, Publication manual of the American Psychological Association,
(Fourth Edition), 1994.
[2] Bandura, A., Social foundations of thought and action: A
social cognitive theory, Englewood Cliffs, NJ: Prentice-Hall,
1986.
[3] Bellini, J., Bolton, B. and Neath, J., Rehabilitation counselors
assessments of applicants functional limitations as predictors
of rehabilitation services provided, Rehabilitation Counseling
Bulletin 41(4) (1998), 242–258.
[4] Bellini, J. and Rumrill, P., Research in rehabilitation counseling: A guide to design, methodology, and utilization, Springfield, IL: Charles C. Thomas Publishers, 1999.
[5] Bolton, B. and Brookings, J., Development of a multifaceted
definition of empowerment, Rehabilitation Counseling Bulletin 39(4) (1996), 256–264.
[6] Diksa, E. and Rogers, E., Employer concerns about hiring persons with psychiatric disability: Results of the employer attitude questionnaire, Rehabilitation Counseling Bulletin 40(1)
(1996), 31–44.
[7] Hayes, W., Statistics for psychologists, New York: Holt, Rinehart, and Winston, 1981.
[8] Heppner, P., Kivlighan, D. and Wampold, B., Research design
in counseling, Pacific Grove, CA: Brooks/Cole, 1992.
[9] Heppner, P., Kivlighan, D. and Wampold, B., Research design
in counseling, (2nd Edition), Pacific Grove, CA: Brooks/Cole,
1999.
[10] Hershenson, D., A systems reformulation of a developmental
model of work adjustment, Rehabilitation Counseling Bulletin
40(1) (1996), 2–10.
[11] Hunter, J., Needed: A ban on the significance test, Psychological Science 8 (1997), 3–7.
[12] Kazdin, A., Research design in clinical psychology, (2nd Edition), Needham Heights, MA: Allyn and Bacon, 1998.
[13] Krause, J. and Anson, C., Self-perceived reasons for unemployment cited by persons with spinal cord injury: Relationship to gender, race, age, and level of injury, Rehabilitation
Counseling Bulletin 39(3) (1996), 217–227.
[14] McClure, P., Determining the significance of significance: Pvalues, effect size, and clinical judgement, Journal of Hand
Therapy 12 (1999), 40–41.
[15] McMillan, J. and Schumacher, S., Research in education: A
conceptual introduction, (Fourth Edition), New York: Longman, 1997.
[16] Munley, P., Sharkin, B. and Gelso, C., Reviewer ratings
and agreement on manuscripts reviewed for the Journal of
Counseling Psychology, Journal of Counseling Psychology 35
(1988), 198–202.
[17] Orwell, G., Politics and the English language, in: A collection
of essays, G. Orwell ed., San Diego: Harcourt, Brace, and
Jovanovich, 1946, pp. 156–171.
[18] Ravesloot, C. and Seekins, T., Vocational rehabilitation counselors’ attitudes toward self-employment outcomes, Rehabilitation Counseling Bulletin 39(3) (1996), 189–201.
[19] Rumrill, P., Roessler, R. and Denny, G., Increasing confidence
in the accommodation request process among persons with
multiple sclerosis: A career maintenance self-efficacy intervention, Journal of Job Placement 13(1) (1997), 5–9.
[20] Rumrill, P., Roessler, R. and Koch, L., Surveying the employment concerns of people with multiple sclerosis: A participatory action research approach, Journal of Vocational Rehabilitation 12(2) (1999), 75–82.
[21] Schaller, J. and Parker, R., Effect of graduate research instruction on perceived research anxiety, research utility, and
confidence in research skills, Rehabilitation Education 11(4)
(1997), 273–287.
[22] Thompson, B., AERA editorial policies regarding statistical
significance testing: Three suggested reforms, Educational
Researcher 25(2) (1996), 26–30.
P. Rumrill et al. / Guidelines for evaluating research articles 261
Table 1
A scale for critiquing research articles
Instructions: Answer the following questions regarding the article, “ ”. Use examples
from the article to support your analyses.
A. Title
1. Does the title describe the study?
2. Do the key words of the title serve as key elements of the article?
3. Is the title concise, i.e., free of distracting or extraneous phrases?
B. Abstract
4. Does the abstract summarize the study’s purpose, methods, and findings?
5. Does the abstract reveal the independent and dependent variables under study?
6. Are there any major premises or findings presented in the article that are not mentioned in the abstract?
7. Does the abstract provide you with sufficient information to determine whether you would be interested in reading
the entire article?
C. Introduction
8. Is the research problem clearly identified?
9. Is the problem significant enough to warrant the study that was conducted?
10. Do the authors present a theoretical rationale for the study?
11. Is the conceptual framework of the study appropriate in light of the research problem?
12. Do the author’s hypotheses and/or research questions seem logical in light of the conceptual framework and research
problem?
13. Are hypotheses and research questions clearly stated? Are they directional?
14. Overall, does the literature review lead logically into the Method section?
D. Method
15. Is the sample clearly described, in terms of size, relevant characteristics, selection and assignment procedures, and
whether any inducements were used to solicit subjects?
16. Do the instruments described seem appropriate as meausres of the variables under study?
17. Have the authors included sufficient information about the psychometric properties (e.g., reliability and validity) of
the instruments?
18. Are the materials used in conducting the study or in collecting data clearly described?
19. Are the study’s scientific procedures thoroughly described in chronological order?
20. Is the design of the study identified (or made evident)?
21. Do the design and procedures seem appropriate in light of the research problem, conceptual framework, and research
questions/hypotheses?
22. Overall, does the method section provide sufficient information to replicate the study?
E. Results
23. Is the Results section clearly written and well organized?
24. Are data coding and analysis appropriate in light of the study’s design and hypotheses?
25. Are salient results connected directly to hypotheses?
26. Are tables and figures clearly labeled? Well organized? Necessary (non-duplicative of text)?
F. Discussion and Conclusion
27. Are the limitations of the study delineated?
28. Are findings discussed in terms of the research problem, conceptual framework, and hypotheses?
29. Are implications for future research and/or rehabilitation counseling practice identified?
30. Are the author’s general conclusions warranted in light of the results?
G. References
31. Is the reference list sufficiently current?
32. Do works cited reflect the breadth of existing literature regarding the topic of the study?
33. Are bibliographic citations used appropriately in the text?
H. General Impressions
34. Is the article well written and organized?
35. Does the study address an important problem in the lives of people with disabilities?
36. What are the most important things you learned from this article?
37. What do you see as the most compelling strengths of this study?
38. How might this study be improved?
Copyright of Work is the property of IOS Press and its content may not be copied or emailed
to multiple sites or posted to a listserv without the copyright holder’s express written
permission. However, users may print, download, or email articles for individual use.

Published by

Dissertations

View all posts