An Initial Development of an Analytic Rubric for Assessing Critical Thinking in English Argumentative Essays of EFL College Students
Article Sidebar
Main Article Content
Abstract
This study aims to initially develop a Critical-Thinking-in-Argumentative-Essay Rubric (CTER) for EFL college students. Participants of this study were five experts and two groups of raters. Data sources included the experts’ validation survey for the CTER, interviews and writing samples. Three phases for developing the CTER were conducted, and the evaluative descriptors of the rubric were revised based on the experts’ comments. To complete the initial development of the rubric, the raters and the first researcher used the CTER to evaluate the writing samples. The scores obtained from the evaluation were analyzed to examine the inter-rater reliability of the rubric. The findings showed that the CTER contained six clear and valid domains for assessing critical thinking in argumentative essays of EFL students. The total scoring results from the six domains achieved a moderate inter-rater reliability with ICC of 0.70 and Kendall’s W of 0.5 (p < 0.05). The raters perceived that the CTER could be used to promote learning and critical thinking of EFL learners. Pedagogical implications were presented based on the findings.
Article Details
References
Adunyarittigun, D. (2015). Developing a scale to measure reader self-perception for EFL students PASAA, 50, 1-30.
Anderson, N. J. (2012). Student involvement in assessment: Healthy self-assessment and effective peer assessment. In C. Coombe, P. Davidson, B. O’Sullivan & S. Stoynoff (Eds.), The Cambridge guide to second language assessment (pp.187-197). Cambridge United Kingdom: Cambridge University Press.
Angeli, C., & Valanides, N. (2009). Instructional effects on critical thinking: Performance on ill- defined issues. Learning and Instruction, 19, 322-334.
Bailey, A., Zanchetta, M., Velasco, D., Pon, G., & Hassan, A. (2015). Building a scholar in writing (BSW): A model for developing students’ critical writing skills. Nurse Education in Practice, 15, 524-529.
Bauer, J. A. (2016). A new approach: Closing the writing gap by using reliable assessment to guide and evaluate cross-curricular argumentative writing (Master’s thesis). University of Wisconsin, Wisconsin, U.S.A.
Brown, G. T. L., Glasswell, K., & Harland, D. (2004). Accuracy in the scoring of writing: Studies of reliability an validity using a New Zealand writing assessment system. Assessing Writing, 9, 105-121.
Castilleja, M. F. P. (2012). Developing a process to create and validate an instrument assessing student attainment of competencies at an intercultural university in Mexico (Unpublished doctoral dissertation). Florida International University, Florida, united States of America.
Celuch, K., & Slama, M. (2010). Teaching critical thinking skills for the 21st century: An advertising principles case study. Journal of Education for Business, 74(3), 134-139.
Chatfield, T. (2018). Critical thinking: Your guide to effective argument, successful analysis and independent study. London, United Kingdom: Sage.
Cheng, F. (2003). The effects of rhetorical specification in writing assignments on EFL (English as a foreign language) writing (Unpublished doctoral dissertation). Boston University, Boston, U.S.A.
Chinda, B. (2013). Consideration in performance-based language assessment: Rating scales and rater training. PASAA, 46, 141-159.
Chinda, B. (2014). Teachers’ reactions towards performance-based language assessment. PASAA, 48, 57-88.
Daud, N. S. B. M. (2012). Developing critical thinking skills in tertiary academic writing through the use of an instructional rubric for peer evaluation (Unpublished doctoral dissertation). The University of Canterbury, Christchurch, New Zealand.
Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). New York, NY: McGraw-Hill.
Dhanarattigannon, J. (1995). Journal writing: Language activity and teaching/learning evaluation. Humanities Journal, 3(1), 69-80.
Dong, Y. (2015). Critical thinking in second language writing: Concept, theory, and pedagogy (Unpublished doctoral dissertation). The University of British Columbia, Vancouver, Canada.
Epstein, R. L. (1999). Critical thinking. Belmont, CA: Wadsworth. Gardner, R. C. (1985). Social psychology and language learning: The role of attitudes and motivation. London, United Kingdom: Edward Arnold.
Gebril, A., & Plakans, L. (2014). Assembling validity evidence for assessing academic writing: Rater reactions to integrated tasks. Assessing Writing, 21, 56-73.
Green, A. (2014). Exploring language assessment and testing: Language in action. New York, NY: Routledge.
Green, A., & Hawkey, R. (2012). Marking assessments: Rating scales and rubrics. In C. A. Coombe, P. Davidson, B. O’Sullivan & S. Stoynoff (Eds.), The Cambridge guide to second language assessment (pp. 299-306). Cambridge, UK: Cambridge University Press.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255-274.
Han, T., & Huang, J. (2017). Examining the impact of scoring methods on the instructional EFL writing assessment: A Turkish perspective. PASAA, 53, 112-147.
Hodges, T. S., Wright, K. L., Winda, S. A., Matthews, S. D., Zimmer, W. K., & McTigue, E. (2019). Developing and examining validity evidence for the Writing Rubric to Inform Teacher Educators (WRITE). Assessing Writing, 40, 1-13.
Hughes, C. (2000). A comparative study of teaching critical thinking through persuasive writing to average, gifted and students with learning disabilities (Unpublished doctoral dissertation). The College of William and Mary, Virginia, U.S.A.
Janssen, G., Meier, V., & Trace, J. (2015). Building a better rubric: Mixed methods rubric revision. Assessing Writing, 26, 51-66.
Johnson, B., & Onwuegbuzie, A. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26.
Kalchayanant, T. (2009). Writing essays. Bangkok, Thailand: Thammasat University Press.
Katz, A. (2012). Linking assessment with instructional aims and learning. In C. A. Coombe, P. Davidson, B. O’Sullivan & S. Stoynoff (Eds.), The Cambridge guide to second language assessment (pp. 66-73). Cambridge, UK: Cambridge University Press.
Kirby, G. R., & Goodpaster, J. R. (2007). Thinking: An interdisciplinary approach to critical and creative thought (4th ed.). Cranbury, NJ: Pearson Education, Inc.
Koo, T. K., & Li, BPS. M. L. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155-163.
Lau, J. Y. F. (2011). An introduction to critical thinking and creativity: Think more, think better. Hoboken, NJ: John Wiley & Sons, Inc.
Liu, F., & Stapleton, P. (2014). Counterargumentation and the cultivation of critical thinking in argumentative writing: Investigating washback from a high-stakes test. System, 45, 117-128.
Liu, F., & Stapleton, P. (2018). Connecting writing assessment with critical thinking: An exploratory study of alternative rhetorical functions and objects of enquiry in writing prompts. Assessing Writing, 38, 10–20.
Liu, P. E., Wu, W. V., & Shieh, R. (2015). Enhancing EFL students’ critical thinking and writing: An asynchronous debate instructional design. English Teaching & Learning, 39(3), 33-59.
Malcolm, B. (2011). Say What?: The use of voice in teaching creative writing. Journal of Letters, Special Issue, 40(2), 104-125.
Manalo, E., & Sheppard, C. (2016). How might language affect critical thinking performance? Thinking Skills and Creativity, 21, 41–49.
Matsuda, P. K., & Hammill, M. J. (2014). Second language writing pedagogy. In G. Tate, A. R. Taggart, K. Schick & H. B. Hessler (Eds.), A guide to composition pedagogies (2nd ed., pp. 266-282). Oxford, United Kingdom: Oxford University Press.
Maybin, J., & Swann, J. (Eds.) (2006). The art of English: Everyday creativity. New York, NY: Palgrave Macmillan.
Mayweg-Paus, E., Thiebach, M., & Jucks, R. (2016). Let me critically question this! Insights from a training study on the role of questioning on argumentative discourse. International Journal of Educational Research, 79, 195-210.
McKitrick, S. A., & Barnes, S. M. (2012). Assessment of critical thinking: An evolutionary approach. Journal of Assessment and Institutional Effectiveness, 2(1), 1-29.
McLaughlin, F., & Moore, M. (2012). Integrating critical thinking into the assessment of college writing. TETYC, 40(2), 145-162.
McLendon, N. C. G. (2008). The effects of teaching critical thinking and reading comprehension strategies on students’ writing in developmental English in a community college (Unpublished doctoral dissertation). Auburn University, Albama, U.S.A.
Merriam, S. B. (1998). Qualitative research and case study applications in education. San Francisco, CA: Jossey-Bass Publishers.
Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241-256.
Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7(10), 1-10.
Mulnix, J. W., & Mulnix, M. J. (2010). Using a writing portfolio project to teach critical thinking skills. Teaching Philosophy, 33(1), 27-54.
Newell, G. E., Beach, R., Smith, J., VanDerHeide, J., Kuhn, D., & Andriessen, J. (2011). Teaching and learning argumentative reading and writing: A review of research. Reading Research Quarterly, 46(3), 273- 304.
Nussbaum, E. M., & Schraw, G. (2007). Promoting argument–counterargument integration in students’ writing. The Journal of Experimental Education, 76(1), 59-92.
Oshima, A., & Hogue, A. (2006). Writing academic English (4th ed.). New York, NY: Pearson Education.
Payutto, P. A. (2010). The language and the truth. Bangkok, Thailand: Chandpen Publishing.
Plakans, L., & Gebril, A. (2017). An assessment perspective on argumentation in writing. Journal of Second Language Writing, 36, 85-86.
Qin, J. (2009). The analysis of Toulmin elements and use of sources in Chinese university EFL argumentative writing (Unpublished doctoral dissertation). Northern Arizona University, Arigona, U.S.A.
Quintieri, M. (2005). Measuring rater consistency: An investigation into the effects of two testing instruments on raters’ scores (Unpublished master’s thesis). Concordia University, Montreal, Canada.
Reznitskaya, A., Kuo, L., Glina, M., & Anderson, R. C. (2009). Measuring argumentative reasoning: What’s behind the numbers? Learning and Individual Differences, 19(2), 219-224.
Rosenwasser, D., & Stephen, J. (2012). Writing analytically (6th ed.). Boston, MA: Wadsworth.
Ross, J. A. (2006). The reliability, validity, and utilization of self-assessment. Practical Assessment, Research & Evaluation, 11(10), 1-13.
Rowicki, M. A. (2001). A study of the relationship between reflective writing and critical thinking in seventh grade integrated science students (Unpublished doctoral dissertation). Auburn University, Albama, U.S.A.
Saxton, E., Belanger, S., & Becker, W. (2012). The critical thinking analytic rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments. Assessing Writing, 17, 251-270.
Schmidt, R. C. (1997). Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences, 28(3), 763-774.
Shepard, L. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.
Stapleton, P. (2001). Assessing critical thinking in the writing of Japanese university students: Insights about assumptions and content familiarity. Written Communication, 18(4), 506-548.
Stapleton, P., & Wu, Y. (2015). Assessing the quality of arguments in students’ persuasive writing: A case study analyzing the relationship between surface structure and substance. Journal of English for Academic Purposes, 7, 12–23.
Stevens, D. D., & Levi, A. (2005). Introduction to rubric: An assessment tool to save grading time, convey effective feedback, and promote student learning. Herndon, VA: Stylus Publishing.
Tangkiengsirisin, S. (2014). Links between teacher written feedback and student revision. LEARN Journal, Special Issue, 50-60.
Tapinta, P. (2006). Exploring Thai EFL university students’ awareness of their knowledge, use, and control of strategies in reading and writing (Unpublished doctoral dissertation). University of North Texas, Texas, U.S.A.
Thongrin, S. (2002). E-mail peer response in collectivist Thai culture: Task, social, and cultural dimensions (Unpublished doctoral dissertation). Indiana University of Pennsylvania, Pennsylvania, U.S.A.
Tittle, P. (2011). Critical thinking: An appeal to reason. New York, NY: Routedge.
Tulgar, A. T., Yagiz, O., & Han, T. (2017). Bridging the gap between linguistic and pragmatic assessment in Turkish contexts. PASAA, 54, 58-81.
Verlinden, J. (2005). Critical thinking and everyday argument. Boston, MA: Wadsworth, Cengage Learning.
Weigle, S. C. (2002). Assessing writing. Cambridge, United Kingdom: Cambridge University Press.
Welch, K. C., Hieb, J., & Graham, J. (2015). A systematic approach to teaching critical thinking skills to electrical and computer engineering undergraduates. American Journal of Engineering Education, 6(2), 113-123.
Williams, R. L. (1999). Operational definitions and assessment of higher-order cognitive constructs. Educational Psychology Review, 11(4), 411-427.
Yu, H. J. (2008). Interactional structure in the writing process: A comparison of three ESL writing classes (Unpublished doctoral dissertation). Arizona State University, Arizona, U.S.A.
Zhao, C. G. (2010). The role of voice in high-stakes second language writing assessment (Unpublished doctoral dissertation). New York University, New York, U.S.A.