Article
| Title: | Measuring production and comprehension of written arguments in upper-elementary grades |
| Author: | |
| Source document: | Studia paedagogica. 2019, vol. 24, iss. 4, pp. [63]-84 |
| Extent: | [63]-84 |
| ISSN: | 1803-7437 (print); 2336-4521 (online) |
| DOI: | 10.5817/SP2019-4-3 |
| Type: | Article (English) |
| Stable URL: | http://hdl.handle.net/11222.digilib/142240 |
| License: |
PDF
Abstract(s):
We describe a systematic process of developing measures of argument production and comprehension. These measures, designed for students in upper-elementary language arts classrooms, are called Writing Argument and Reading Argument. We discuss the rationale and theoretical framework for the measures, describe pilot and validation studies, and present initial findings to support the reliability, validity, and usability of these measures. Our results showed that both measures had acceptable inter-rater reliability. The correlations among Writing Argument, Reading Argument and an established reading comprehension test were moderate, which highlights the importance of task-specific competencies. The performance on both measures was not associated with ethnicity of the students. Gender was a significant predictor, with girls performing better than boys. Teachers found both measures to be pedagogically useful. Although some teachers initially struggled with learning how to use the scoring rubrics, they generally found the scoring for both tasks to be informative for their practice.
Keywords:
writing argument; reading argument; measures; validation; elementary grades
References:
[1] Anderson, R. C., Nguyen-Jahiel, K., McNurlen, B., Archodidou, A., Kim, S.-y., Reznitskaya, A., Tillmanns, M., & Gilbert, L. (2001). The snowball phenomenon: Spread of ways of talking and ways of thinking across groups of children. Cognition and Instruction, 19(1), 1–46. | DOI 10.1207/S1532690XCI1901_1
[2] Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading comprehension. In P. D. Pearson, R. Barr, M. L. Kamil & P. Mosenthal (Eds.), Handbook of reading research (pp. 255–291). New York: Longman.
[3] Asher, S. R., Hymel, S., & Wigfield, A. (1978). Influence of topic interest on children's reading comprehension. Journal of Reading Behavior, 10(1), 35–47. | DOI 10.1080/10862967809547253
[4] Chambliss, M. J., & Murphy, P. K. (2002). Fourth and fifth graders representing the argument structure in written texts. Discourse Processes, 34(1), 91–115.
[5] Chinn, A. C., Duncan, R. G., Hung, L. C.-C., & Rinehart, R. W. (2016). Epistemic criteria and reliable processes as indicators of argument quality in science students' argumentation. Paper presented at the Annual Meeting of the American Educational Research Association, Washington, DC.
[6] Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont: Wadsworth.
[7] Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation in classrooms. Science Education, 84(3), 287–312. | DOI 10.1002/(SICI)1098-237X(200005)84:3<287::AID-SCE1>3.0.CO;2-A
[8] Ennis, R. (1996). Critical thinking. Upper Saddle River: Prentice Hall.
[9] Ennis, R. (2003). Critical thinking assessment. In D. Fasko (Ed.), Critical thinking and reasoning (pp. 293–313). Cresskill: Hampton Press, Inc.
[10] Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of toulmin's argument pattern for studying science discourse. Science Education, 88(6), 915–933. | DOI 10.1002/sce.20012
[11] Ferretti, R. P., & Fan, Y. (2016). Argumentative writing. In C. A. MacArthur, S. Graham & J. Fitzgerald (Eds.), Handbook of writing research, Second edition. New York: Guilford Press.
[12] Fischer, F., Chinn, C. A., Engelmann, K., & Osborne, J. (Eds.). (2018). Scientific reasoning and argumentation: The roles of domain-specific and domain-general knowledge. New York: Routledge.
[13] Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive psychology, 15(1), 1–38. | DOI 10.1016/0010-0285(83)90002-6
[14] Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., Lee, C.D., Shanahan, C., & Project, R. (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), 219–246. | DOI 10.1080/00461520.2016.1168741
[15] Govier, T. (1987). Problems in argument analysis and evaluation. Providence: Foris.
[16] Govier, T. (2010). A practical study of argument. Belmont: Wadsworth Publishing Company.
[17] Graff, G. (2003). Clueless in academe. New Haven: Yale University Press.
[18] Halpern, D. F. (2003). The "how" and "why" of critical thinking assessment. In D. Fasko (Ed.), Critical thinking and reasoning (pp. 331–354). Cresskill: Hampton Press, Inc.
[19] Healy, P. (1987). Critical reasoning and dialectical argument: An extension of Toulmin's approach. Informal Logic, 9(1), 1–12. | DOI 10.22329/il.v9i1.2654
[20] Hidi, S. (2001). Interest, reading, and learning: Theoretical and practical considerations. Educational Psychology Review, 13(3), 191–209.
[21] Hollihan, T. A., & Baaske, K. T. (1973). Arguments and arguing: The products of human decision making. Prospect Hights: Waveland.
[22] Hughes, J. N. (1992). Review of the Cornell critical thinking tests. In J. J. Kramer, J. C. Conoley & L. L. Murphy (Eds.), The eleventh mental measurements yearbook. Lincoln: The Buros Institute of Mental Measurements.
[23] Johnson, K. M., (2005). Gates-MacGinitie reading tests fourth edition forms S and T. In B. S. Plake & R. A. Spies (Eds.), The sixteenth mental measurements yearbook (Vol. 16). Lincoln: Buros Institute of Mental Measurements.
[24] Klahr, D., Zimmerman, C., & Jirout, J. (2011). Educational interventions to advance children's scientific thinking. Science, 333(6045), 971–975. | DOI 10.1126/science.1204528
[25] Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. | DOI 10.1016/j.jcm.2016.02.012
[26] Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press.
[27] Kuhn, D. (2005). Education for thinking. Boston: Harvard Education Press.
[28] Kuhn, D., & Crowell, A. (2011). Dialogic argumentation as a vehicle for developing young adolescents' thinking. Psychological Science, 22(4), 545–552. | DOI 10.1177/0956797611402512
[29] Kuhn, D., Hemberger, L., & Khait, V. (2016). Argue with me: Argument as a path to developing students' thinking and writing. New York: Routledge.
[30] Lipman, M. (2003). Thinking in education. New York: Cambridge University Press.
[31] MacGinitie, W. H., MacGinitie, R. K., Maria, K. & Dreyer, L. G. (2002). Gates-MacGinitie reading tests. Rolling Meadows: Riverside Publishing.
[32] Mason, L., & Scirica, F. (2006). Prediction of students' argumentation skills about controversial topics by epistemological understanding. Learning and Instruction, 16(5), 492–509. | DOI 10.1016/j.learninstruc.2006.09.007
[33] McCabe, P. P. (2005). Review of the Gates-MacGinitie reading tests. In R. A. Spies & B. S. Plake (Eds.), The Sixteenth Mental Measurements Yearbook. Lincoln: Buros Institute of Mental Measurements.
[34] Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal reasoning among children of different grade, ability, and knowledge levels. Cognition and Instruction, 14(2), 139–178. | DOI 10.1207/s1532690xci1402_1
[35] Mercer, N. (2011). Reasoning serves argumentation in children. Cognitive Development, 26(3), 177–191. | DOI 10.1016/j.cogdev.2010.12.001
[36] National Governors Association. (2010). Common core state standards: Appendix A. research supporting key elements of the standards. Washington DC: National Governors Association Center for Best Practices, Council of Chief State School Officers.
[37] Newell, G. E., Beach, R., Smith, J., VanDerHeide, J., Kuhn, D., & Andriessen, J. (2011). Teaching and learning argumentative reading and writing: A review of research. Reading Research Quarterly, 46(3), 273–304.
[38] Nielsen, J. A. (2013). Dialectical features of students' argumentation: A critical review of argumentation studies in science education. Research in Science Education, 43(1), 371–393. | DOI 10.1007/s11165-011-9266-x
[39] Norris, S. P. (1995). Format effects on critical thinking test performance. The Alberta Journal of Educational Research, 41(4), 378–406.
[40] Norris, S. P., Leighton, J. P., & Phillips, L. M. (2004). What is at stake in knowing the content and capabilities of children's minds? A case for basing high stakes tests on cognitive models. Theory & Research in Social Education, 2(3), 283–308. | DOI 10.1177/1477878504046524
[41] Nussbaum, E. M. (2011). Argumentation, dialogue theory, and probability modeling: Alternative frameworks for argumentation research in education. Educational Psychologist, 46(2), 84–106. | DOI 10.1080/00461520.2011.558816
[42] Nussbaum, E. M., & Bendixen, L. D. (2003). Approaching and avoiding arguments: The role of epistemological beliefs, need for cognition, and extraverted personality traits. Contemporary Educational Psychology, 28(4), 573–595. | DOI 10.1016/S0361-476X(02)00062-0
[43] Nussbaum, E. M., & Kardash, C. M. (2005). The effects of goal instruction and text on the generation of counterarguments during writing. Journal of Educational Psychology, 9(2), 157–169. | DOI 10.1037/0022-0663.97.2.157
[44] Nussbaum, E. M., & Ordene, V. (2011). Critical questions and argument stratagems: A framework for enhancing and analyzing students' reasoning practices The Journal of the Learning Sciences, 20(3), 443–488. | DOI 10.1080/10508406.2011.564567
[45] Oka, E. R., & Paris, S. G. (1987). Patterns of motivation and reading skill in underachieving children. In S. J. Ceci (Ed.), Handbook of cognitive, social, and neuropsychological aspects of learning disabilities (pp. 115–145). New Jersey: Lawrence Erlbaum Associates.
[46] Page-Voth, V., & Graham, S. (1999). Effects of goal setting and strategy use on the writing performance and self-efficacy of students with writing and learning problems. Journal of Educational Psychology, 91(2), 230–240. | DOI 10.1037/0022-0663.91.2.230
[47] Partnership for 21st Century Skills. (2012). A Framework for 21st Century Learning. Retrieved from http://www.p21.org/index.php
[48] Phillips, L. M., & Patterson, C. C. (1987). Test of inference ability in reading comprehension. Newfoundland: Institute for Educational Research and Development.
[49] Poteet, J. (1989). Review of the Ennis-Weir critical thinking essay test. J. C. Conoley & J. J. Kramer (Eds.), The tenth mental measurements yearbook (pp. 289–290). Lincoln: The Buros Institute of Mental Measurements.
[50] Rapanta, C., Garcia-Mila, M., & Gilabert, S. (2013). What is meant by argumentative competence? An integrative review of methods of analysis and assessment in education. Review of Educational Research, 83(4), 483–520. | DOI 10.3102/0034654313487606
[51] Renninger, K. A., Suzanne Hidi, & Krapp, A. (Eds.). (1992). The role of interest in learning and development. Mahwah: Lawrence Erlbaum Associates.
[52] Reznitskaya, A., Anderson, R. C., Dong, T., Li, Y., Kim, I., & Kim, S. (2008). Learning to think well: Application of argument schema theory. In C. C. Block & S. Parris (Eds.), Comprehension instruction: Research-based best practices (pp. 196–213). New York: Guilford Press.
[53] Reznitskaya, A., Anderson, R. C., McNurlen, B., Nguyen-Jahiel, K., Archodidou, A., & Kim, S. (2001). Influence of oral discussion on written argument. Discourse Processes, 32(2 & 3), 155–175.
[54] Reznitskaya, A., & Gregory, M. (2013). Student thought and classroom language: Examining the mechanisms of change in dialogic teaching. Educational Psychologist, 48(2), 114–133. | DOI 10.1080/00461520.2013.775898
[55] Reznitskaya, A., Kuo, L., Clark, A., Miller, B., Jadallah, M., Anderson, R. C., & Nguyen-Jahiel, K. (2009). Collaborative reasoning: A dialogic approach to group discussions. Cambridge Journal of Education, 39(1), 29–48. | DOI 10.1080/03057640802701952
[56] Reznitskaya, A., Kuo, L., Glina, M., & Anderson, R. C. (2009). Measuring argumentative reasoning: What's behind the numbers? Learning and individual differences, 19(2), 219–224. | DOI 10.1016/j.lindif.2008.11.001
[57] Reznitskaya, A., & Wilkinson, I. A. G. (2015). Professional development in dialogic teaching: Helping teachers promote argument literacy in their classrooms. In D. Scott & E. Hargreaves (Eds.), Sage handbook of learning (pp. 219–232). London: Sage Publications.
[58] Reznitskaya, A., & Wilkinson, I. A. G. (2017). The most reasonable answer: Helping students build better arguments together. Boston: Harvard Education Press.
[59] Rumelhart, D. E., & Ortony, A. (1977). The representation of knowledge in memory. In R. C. Anderson, R. J. Spiro, & W. E. Montague (Eds.), Schooling and the acquisition of knowledge (pp. 99–136). Hillsdale: Erlbaum.
[60] Shepard, L. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 1–14. | DOI 10.3102/0013189X029007004
[61] Stein, N. L., & Trabasso, T. (1982). Children's understanding of stories: A basis for moral judgment and dilemma resolution. In C. J. Brainerd & M. Pressley (Eds.), Verbal processes in children: Progress in cognitive development research (pp. 161–188). New York: Springer-Verlag.
[62] Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4), 1–19.
[63] Sutton, R. E. (1992). Review of the New Jersey test of reasoning skills. In J. J. Kramer, J. C. Conoley & L. L. Murphy (Eds.), The eleventh mental measurements yearbook (pp. 606–608). Lincoln: The Buros Institute of Mental Measurements.
[64] Toulmin, S. E. (1958). The uses of argument. Cambridge: Cambridge University Press.
[65] Walton, D. (1998). The new dialectic: Conversational contexts of argument. Toronto: University of Toronto Press.
[66] Weinstock, M. P., Neuman, Y., & Glassner, A. (2006). Identification of informal reasoning fallacies as a function of epistemological level, grade level, and cognitive ability. Journal of Educational Psychology, 98(2), 327–341. | DOI 10.1037/0022-0663.89.2.327
[67] Wilkinson, I. A. G., Reznitskaya, A., Bourdage, K., Oyler, J., Nelson, K., Glina, M., Drewry, R., Kim, M.-Y. (2017). Toward a more dialogic pedagogy: Changing teachers' beliefs and practices through professional development in language arts classrooms. Language & Education, 31(1), 65–82. | DOI 10.1080/09500782.2016.1230129
[68] Wolfe, C. R., Britt, M. A., & Butler, J. A. (2009). Argumentation schema and the my side bias in written argumentation. Written Communication, 26(2), 183–209. | DOI 10.1177/0741088309333019
