Systematic Literature Reviews in Engineering Education andOtherDeveloping Interdisciplinary Fields


In fields such as medicine, psychology, and education, systematic reviews of the literature critically appraise and summarize research to inform policy and practice. We argue that now is an appropriate time in the development of the field of engineering education to both support systematic reviews and benefit from them. More reviews of prior work conducted more systematically would help advance the field by lowering the barrier for both researchers and practitioners to access the literature, enabling more objective critique of past efforts, identifying gaps, and proposing new directions for research.


Andrews, R. (2005). The place of systematic reviews in education research. British Journal of
Educational Studies, 53(4), 399–416.
Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: A criti-
cal review. BMC Medical Research Methodology, 9(59), 1–10. doi: 10.1186/1471-2288-9-59
Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews. Review of
General Psychology, 1(3), 311–320.
Becker, K., & Park, K. (2011). Effects of integrative approaches among science, technology,
engineering, and mathematics (STEM) subjects on students’ learning: A preliminary
meta-analysis. Journal of STEM Education: Innovations & Research, 12(5), 23–37.
Beddoes, K., & Borrego, M. (2011). Feminist theory in three engineering education jour-
nals: 1995–2008. Journal of Engineering Education, 100(2), 281–303.
Beddoes, K., Jesiek, B. K., & Borrego, M. (2010). Identifying opportunities for collabora-
tions in international engineering education research on problem- and project-based
learning. Interdisciplinary Journal of Problem-based Learning, 4(2), 7–34.

Benitti, F. B. V. (2012). Exploring the educational potential of robotics in schools: A sys-
tematic review. Computers & Education, 58(3), 978–988.
Berlin, J. A. (1997). Does blinding of readers affect the results of meta-analyses? Lancet,
350(9072), 185–186.
Borrego, M. (2007a). Conceptual difficulties experienced by engineering faculty becoming
engineering education researchers. Journal of Engineering Education, 96(2), 91–102.
Borrego, M. (2007b). Development of engineering education as a rigorous discipline: A
study of the publication patterns of four coalitions. Journal of Engineering Education,
96(1), 5–18.
research methods in engineering education. Journal of Engineering Education, 98(1), 53–66.
Borrego, M., Froyd, J. E., & Hall, T. S. (2010). Diffusion of engineering education innova-
tions: A survey of awareness and adoption rates in U.S. engineering departments. Journal
of Engineering Education, 99(3), 185–207.
Borrego, M., & Newswander, L. K. (2008). Characteristics of successful cross-disciplinary
engineering education collaborations. Journal of Engineering Education, 97(2), 123–134.
Borrego, M., Streveler, R. A., Miller, R. L., & Smith, K. A. (2008). A new paradigm for a
new field: Communicating representations of engineering education research. Journal of
Engineering Education, 97(2), 147–162.
Campbell Collaboration. (n.d.). About Us [Web page]. Retrieved from http://www.
Cook, D. A., & West, C. P. (2012). Conducting systematic reviews in medical education: A
stepwise approach. Medical Education, 46, 943–952.
Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997). Systematic reviews: Synthesis of best
evidence for clinical decisions. Annals of Internal Medicine, 126(5), 376–380.
Crismond, D. P., & Adams, R. S. (2012). The informed design teaching and learning
matrix. Journal of Engineering Education, 101(4), 738–797.
Downe-Wamboldt, B. (1992). Content analysis: Methods, applications and issues. Health
Care for Women International, 13, 313–321.
Durand, M.-A., Stiel, M., Boivin, J., & Elwyn, G. (2008). Where is the theory? Evaluating
the theoretical frameworks described in decision support technologies. Patient Education
and Counseling, 71(1), 125–135. doi: 10.1016/j.pec.2007.12.004
Eva, K. M. (2008). On the limits of systematicity. Medical Education, 42(9), 852–853. doi:
Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre).
(2010). EPPI-Centre methods for conducting systematic reviews. London: University of
Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A
meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3),
Fensham, P. J. (2004). Defining an identity: The evolution of science education as a field of
research. New York, NY: Springer.
Froyd, J. E., & Ohland, M. W. (2005). Integrated engineering curricula. Journal of Engi-
neering Education, 94(1), 147–164.
Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction (8th
ed.). Boston, MA: Pearson.Gough, D. (2004). Systematic research synthesis. In G. Thomas & R. Pring (Eds.), Evi-
dence-based practice in education (pp. 44–62). Berkshire, UK: Open University Press.
Gough, D., Oliver, S., & Thomas, J. (2012). An introduction to systematic reviews. Los
Angeles, CA: Sage.
Graneheim, U. H., & Lundman, B. (2004). Qualitative content analysis in nursing research:
Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today,
24, 105–112.
Haghighi, K. (2005). Quiet no longer: Birth of a new discipline. Journal of Engineering Edu-
cation, 94(4), 351–353.
Henderson, C., Finkelstein, N. D., & Beach, A. (2010). Beyond dissemination in college
science teaching: An introduction to four core change strategies. Journal of College Science
Teaching, 39(5), 18–25.
Heywood, J. (2005). Engineering education: Research and development in curriculum and
instruction. Hoboken, NJ: Wiley-IEEE Press.
Higgins, J. P. T., & Green, S. (2009). Cochrane handbook for systematic reviews of interven-
tions (Cochrane Collaboration Ed.). Hoboken, NJ: Wiley-Blackwell.
Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis.
Qualitative Health Research, 15(9), 1277–1288.
Institute of Education Sciences. (2012). What Works Clearinghouse. Retrieved from http://
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J., Gavaghan, D. J., &
McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is
blinding necessary? Controlled Clinical Trials, 17(1), 1–12.
Jesiek, B. K., Borrego, M., Beddoes, K., Hurtado, M., Rajendran, P., & Sangam, D. (2011).
Mapping global trends in engineering education research, 2005–2008. International Jour-
nal of Engineering Education 27(1), 77–90.
Jesiek, B. K., Newswander, L. K., & Borrego, M. (2009). Engineering education research:
Discipline, community, or field? Journal of Engineering Education, 98(1), 39–52.
Johnson, D. W., Johnson, R. T., & Smith, K. A. (1991). Active learning: Cooperation in the
college classroom (3rd ed.). Edina, MN: Interaction Book Company.
Johnson, D. W., Johnson, R. T., & Smith, K. A. (1998). Cooperative learning returns to
college: What evidence is there that it works? Change, 30(4), 26–35.
Johri, A., & Olds, B. (2011). Situated engineering learning: Bridging engineering education
research and the learning sciences. Journal of Engineering Education, 100(1), 151–185.
Johri, A., & Olds, B. (Eds.). (in press). Cambridge handbook of engineering education research.
Kiefer, M., & Pulverm€ uller, F. (2012). Conceptual representations in mind and brain: The-
oretical developments, current evidence and future directions. Cortex, 48(7), 805–825.
doi: 10.1016/j.cortex.2011.04.006
Kondracki, N. L., Wellman, N. S., & Amundson, D. R. (2002). Content analysis: Review
of methods and their applications in nutrition education. Journal of Nutrition Education
and Behaviour, 34(4), 224–230.
Koro-Ljungberg, M., & Douglas, E. P. (2008). State of qualitative research in engineering
education: Meta-analysis of JEE articles, 2005–2006. Journal of Engineering Education,
97(2), 163–176.
Leedy, P. D., & Ormrod, J. E. (2005). Practical research: Planning and design. Upper Saddle
River, NJ: Merrill.Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school
students’ mathematics learning. Educational Psychology Review, 22(3), 215–243.
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P., . . .
Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-
analyses of studies that evaluate healthcare interventions: Explanation and elaboration.
BMJ (Clinical Research Education), 339, b2700. doi: 10.1136/bmj.b2700
Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cam-
bridge, MA: Harvard University Press.
Litzinger, T., Lattuca, L. R., Hadgraft, R., & Newstetter, W. (2011). Engineering educa-
tion and the development of expertise. Journal of Engineering Education, 100(1), 123–150.
Lohmann, J. R., & Froyd, J. E. (2010). Chronological and ontological development of engineer-
ing education as a field of scientific inquiry. Paper presented at the Second Meeting of the
Committee on the Status, Contributions, and Future Directions of Discipline-Based
Education Research, Washington DC.
Lou, Y. (2004). Understanding process and affective factors in small group versus individual
learning with technology. Journal of Educational Computing Research, 31(4), 337–369.
Lucena, J., Downey, G., Jesiek, B. K., & Elber, S. (2008). Competencies beyond countries:
The re-organization of engineering education in the United States, Europe, and Latin
America. Journal of Engineering Education, 97(4), 433–447.
McGowan, J., & Sampson, M. (2005). Systematic reviews need systematic searchers. Journal
of the Medical Library Association, 93(1), 74–80.
Meese, N., & McMahon, C. (2012). Knowledge sharing for sustainable development in civil
engineering: A systematic review. AI & Society, 27(4), 437–449.
Mehalik, M., & Schunn, C. (2007). What constitutes good design? A review of empirical
studies of design processes. International Journal of Engineering Education, 22(3), 519.
Monasor, M. J., Vizca ıno, A., Piattini, M., & Caballero, I. (2010). Preparing students and
engineers for global software development: A systematic review. Paper presented at the 5th
IEEE International Conference on Global Software Engineering, Princeton, NJ.
Montfort, D., Brown, S., & Pegg, J. (2012). The adoption of a capstone assessment instru-
ment. Journal of Engineering Education, 101(4), 657–678.
Mullen, B., Driskell, J. E., & Salas, E. (1998). Meta-analysis and the study of group dynam-
ics. Group Dynamics, 2(4), 213–229.
Mullen, P. D., & Ramirez, G. (2006). The promise and pitfalls of systematic reviews.
Annual Review of Public Health, 27, 81–102. doi: 10.1146/annurev.publhealth.27.
National Institute for Health and Clinical Excellence (NICE). (2009). Methods for the devel-
opment of NICE public health guidance (2nd ed.). London: Author.
National Research Council. (2011) Promising practices in undergraduate science, technology,
engineering, and mathematics education: Summary of two workshops. Washington, DC:
National Academies Press.
Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography: Synthesizing qualitative studies.
London: Sage.
Osorio, N., & Osorio, M. (2002). Engineering education in Europe and the USA: An anal-
ysis of two journals. Science and Technology Libraries, 23(1), 49–70.
Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature search-
ing for social science systematic reviews: Consideration of a range of search techniques.Health Information & Libraries Journal, 27(2), 114–122. doi: 2048/10.1111/j.1471–
Petticrew, M., & Roberts, H. (2002). Evidence, hierarchies, and typologies: Horses for
courses. Journal of Epidemiology and Community Health, 57, 527–529.
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide.
Malden, MA: Blackwell.
Prince, M. J. (2004). Does active learning work? A review of the research. Journal of Engi-
neering Education, 93(3), 223–231. doi: 10.1002/j.2168-9830.2004.tb00809.x
Prince, M. J., & Felder, R. M. (2006). Inductive teaching and learning methods: Defini-
tions, comparisons, and research bases. Journal of Engineering Education, 95(2), 123–128.
doi: 10.1002/j.2168-9830.2006.tb00884.x
Salleh, N., Mendes, E., & Grundy, J. (2011). Empirical studies of pair programming for
CS/SE teaching in higher education: A systematic literature review. IEEE Transactions
on Software Engineering, 37(4), 509–525.
Lowerison, G. (2009). Technology’s effect on achievement in higher education: A Stage I meta-
analysis of classroom applications. Journal of Computing in Higher Education, 21(2), 95–109.
Serial Solutions. (n.d.). Ulrichsweb [Web page]. Retreived from http://www.serialssolutions.
Shavelson, R., & Towne, L. (2002). Scientific research in education. Washington, D.C.:
National Academies Press.
Slavin, R. E. (1984). Meta-analysis in education: How has it been used? Educational
Researcher, 13(8), 6–15.
Springer, L., Stanne, M., & Donovan, S. (1999). Effects of small group learning on under-
graduates in science, mathematics, engineering and technology: A meta-analysis. Review
of Educational Research, 69(1), 21–52.
Streveler, R. A., Borrego, M., & Smith, K. A. (2007). Moving from the scholarship of
teaching and learning to educational research: An example from engineering. In D. R.
Robertson & L. B. Nilson (Eds.), To Improve the Academy (Vol. 25, pp. 139–149). Bos-
ton: Anker.
Streveler, R. A., Litzinger, T. A., Miller, R. L., & Steif, P. S. (2008). Learning conceptual
knowledge in the engineering sciences: Overview and future research directions. Journal
of Engineering Education, 97(3), 279–294.
Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research
in systematic reviews. BMC Medical Research Methodology, 8(45), 1–10. doi: 10.1186/
Thompson, L. A. (2001). Grey literature in engineering. Science & Technology Libraries,
19(3-4), 57–73. doi: 10.1300/J122v19n03_05
Tondeur, J., Van Braak, J., Sang, G., Voogt, J., Fisser, P., & Ottenbreit-Leftwich, A.
(2012). Preparing pre-service teachers to integrate technology in education: A synthesis
of qualitative evidence. Computers & Education, 59(1), 134–144.
Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing
evidence-informed management knowledge by means of systematic review. British Journal
of Management, 14(3), 207–222. doi: 10.1111/1467-8551.00375
Tricco, A. C., Tetzlaff, J., & Moher, D. (2011). The art and science of knowledge synthesis.
Journal of Clinical Epidemiology, 64(1), 11–20.U.S. Department of Education. (2007). Report of the Academic Competitiveness Council.
Washington, DC: Author.
Wankat, P. C. (1999). An analysis of the articles in the Journal of Engineering Education.
Journal of Engineering Education, 88(1), 37–42.
Wankat, P. C. (2004). Analysis of the first ten years of the Journal of Engineering Educa-
tion. Journal of Engineering Education, 93(1), 13–21.
Wankat, P. C., Felder, R. M., Smith, K. A., & Oreovicz, F. S. (2002). The scholarship of
teaching and learning in engineering. In M. T. Huber & S. P. Morreale (Eds.), Discipli-
nary styles in the scholarship of teaching and learning: Exploring common ground (pp. 217–
237). Sterling, VA: Stylus.
What Works Clearinghouse. (2011). Procedures and standards handbook. Washington, DC:
U.S. Department of Education, Institute of Education Sciences.
Whitin, K., & Sheppard, S. (2004). Taking stock: An analysis of the publishing record as
represented by the Journal of Engineering Education. Journal of Engineering Education,
93(1), 5–12.