Development of a Multidimensional Computerized Adaptive Testing Program for Measuring Digital Competency Level in Higher Education
Article Sidebar
Main Article Content
Abstract
Background and Objectives: Thailand has implemented policies to promote and develop digital competencies among students, which require all universities to assess students’ digital competencies prior to graduation. As a result, many small universities face resources shortages for conducting such assessments. Therefore, the development of an online testing system serves as a potential solution to this issue. This research aimed to develop a digital competency item bank and validate the effectiveness of a Multidimensional Computerized Adaptive Testing (MCAT) program.
Methodology: This research employed a research and development (R&D) design. Participants consisted of fourth-year undergraduate students enrolled in higher education institutions in Thailand during the academic year 2023–2024, from the fields of Health Sciences, Science and Technology, and Humanities and Social Sciences. The results indicated that the digital competency item bank contained 211 items and 30 indicators. The item bank was developed using a three-parameter Multidimensional Item Response Theory (MIRT) model. The test demonstrated an EAP reliability of 0.865. Digital competency levels were interpreted using a Wright Map. Additionally, the MCAT program was developed using PHP and MySQL.
Results: The quality of 211 items in the item bank was analyzed using three-parameter item response theory. The items demonstrated an average discrimination parameter of 1.44, an average difficulty parameter of 0.24, and an average guessing parameter of 0.13. The item bank was categorized into three modules based on the difficulty parameter. The digital competency model was developed in accordance with the between-items multidimensional item response theory with three parameters. Digital competency level cutoff scores were determined by defining a criterion zone on the Wright map at the mean ability threshold (θd) of -0.39. The results indicated that 69.27 percent of students met the digital competency criteria. The highest level of digital competency was observed in Component 1, Information and Digital Media Literacy, followed by Component 3, Adaptability to the Digital Transformation Era, while the lowest level was found in Component 2, Skill for using, developing, and problem-solving with digital tools. In addition, the findings demonstrated that the MCAT program accurately evaluated digital competency with only 22 items, using a variable-length termination criterion of SE(θ) < 0.30 (RMSE = 0.112). This approach provides an effective method to reduce the testing burden.
Discussions: The MCAT program demonstrates the feasibility of implementing a reliable and efficient assessment tool in resource-constrained higher education institutions. At the same time, the study suggests a direction for future development in competency assessment on a web-based platform, which enhances the possibilities for implementing MIRT within an educational organization.
Conclusions: The MCAT program demonstrated strong performance across all assessed dimensions, including content validity, indicators, and program structure. The MCAT program consists of an item bank, initial item selection, subsequent item selection, and a termination criterion. The MCAT approach requires less time and fewer items than traditional tests, with the optimal number of items for accurate ability estimation being 22. Overall, the MCAT program provides an appropriate and practical solution for equitable implementation across large, medium, and small universities.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
Babcock, B., & Weiss, D. J. (2012). Termination criteria in computerized adaptive tests: Do variable-length CATs provide efficient and effective measurement? Journal of Computerized Adaptive Testing, 1(1), 1-18. https://doi.org/10.7333/1212-0101001
Baker, F. B., & Kim, S. H. (2017). The basics of item response theory using R (Vol. 10). Springer. https://doi.org/10.1007/978-3-319-54205-8
Briggs, D. C., & Wilson, M. (2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4(1), 87-100.
Brossman, B. G., & Guille, R. A. (2014). A comparison of multi-stage and linear test designs for medium-size licensure and certification examinations. Journal of Computerized Adaptive Testing, 2(1-4), 18-36.
Chen, S. K., Hou, L., & Dodd, B. G. (1998). A comparison of maximum likelihood estimation and expected a posteriori estimation in CAT using the partial credit model. Educational and Psychological Measurement, 58(4), 569-595. https://doi.org/10.1177/0013164498058004002
Eggen, T. J. H. M., & Straetmans, G. J. J. M. (2000). Computerized adaptive testing for classifying examinees into three categories. Educational and Psychological measurement, 60(5), 713-734. https://doi.org/10.1177/00131640021970862
Elliott, G. (2004). Global business information technology: An integrated systems approach. Pearson Education.
Embretson, S. E., & Reise, S. P. (2013). Item response theory for psychologists. Psychology Press. https://doi.org/10.4324/9781410605269
Finch, H. (2010). Item parameter estimation for the MIRT model: Bias and precision of confirmatory factor analysis—based models. Applied Psychological Measurement, 34(1), 10-26. https://doi.org/10.1177/0146621609336112
Fraumeni, B. M., & Liu, G. (2021). Summary of World Economic Forum,“The Global Human Capital Report 2017—Preparing people for the future of work”. In Measuring human capital (pp. 125-138). Academic Press. https://doi.org/10.1016/B978-0-12-819057-9.00008-1
Frey, A., & Seitz, N. N. (2009). Multidimensional adaptive testing in educational and psychological measurement: Current state and future challenges. Studies in Educational Evaluation, 35(2-3), 89-94. https://doi.org/10.1016/j.stueduc.2009.10.007
Hambleton, R. K., & Swaminathan, H. (1985). Estimation of item and ability parameters. In Item response theory: Principles and applications (pp. 125-150). Springer. https://doi.org/10.1007/978-94-017-1988-9_7
Hodson, T. O. (2022). Root mean square error (RMSE) or mean absolute error (MAE): When to use them or not. Geoscientific Model Development Discussions, 15(4), 5481-5487. https://doi.org/10.5194/gmd-15-5481-2022
Homjan, S., Thayreakum, S., & Kumpon, S. (2016). The efficiency of a Computerized Adaptive Testing (CAT) by applying multidimensional item response models. Social Sciences Research and Academic Journal, 11(31), 73–88.
Junpeng, P., Inprasitha, M., & Wilson, M. (2018). Modeling of the open-ended items for assessing multiple proficiencies in mathematical problem solving. The Turkish Online Journal of Educational Technology, 2, 142-149.
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into practice, 41(4), 212-218. https://doi.org/10.1207/s15430421tip4104_2
Li, J., Sun, J., Shao, M., Lai, Y., & Chen, C. (2025). A new multidimensional computerized testing approach: On-the-fly assembled multistage adaptive testing based on multidimensional item response theory. Mathematics, 13(4), 594. https://doi.org/10.3390/math13040594
Li, Y., & Ranieri, M. (2010). Are ‘digital natives’ really digitally competent?—A study on Chinese teenagers. British Journal of Educational Technology, 41(6), 1029-1042. https://doi.org/10.1111/j.1467-8535.2009.01053.x
Lord, F. M., & Novick, M. R. (2008). Statistical theories of mental test scores. IAP.
Lunz, M. E., Wright, B. D., & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3(4), 331-345. https://doi.org/10.1207/s15324818ame0304_3
Piton-Gonçalves, J., & Aluísio, S. M. (2012, October). An architecture for multidimensional computer adaptive test with educational purposes. In Proceedings of the 18th Brazilian symposium on Multimedia and the web (pp. 17-24). Association for Computing Machinery. https://doi.org/10.1145/2382636.2382644
Reckase, M. D. (2009). Historical background for multidimensional item response theory (MIRT). In Multidimensional item response theory (pp. 57-77). Springer. https://doi.org/10.1007/978-0-387-89976-3
Sahin, A., & Weiss, D. J. (2015). Effects of calibration sample size and item bank size on ability estimation in computerized adaptive testing. Educational Sciences: Theory and Practice, 15(6), 1585-1595.
Shao, M., Sun, J., Li, J., Wang, S., & Lai, Y. (2024). Multidimensional hybrid computerized adaptive testing based on multidimensional item response theory. IEEE Access, 169079-169101. https://doi.org/10.1109/ACCESS.2024.3492188
Techataweewan, W., & Prasertsin, U. (2018). Development of digital literacy indicators for Thai undergraduate students using mixed method research. Kasetsart Journal of Social Sciences, 39(2), 215-221. https://doi.org/10.1016/j.kjss.2017.07.001
Thompson, N. A., & Weiss, D. A. (2011). A framework for the development of computerized adaptive tests. Practical Assessment, Research, and Evaluation, 16(1).
Vuorikari, R., Punie, Y., Gomez, S. C., & Van Den Brande, G. (2016). DigComp 2.0: The digital competence framework for citizens. Update phase 1: The conceptual reference model (No. JRC101254). Publication Office of the European Union.
Wang, C., & Chang, H. H. (2011). Item selection in multidimensional computerized adaptive testing—gaining information from different angles. Psychometrika, 76(3), 363-384. https://doi.org/10.1007/s11336-011-9215-7
Wang, C., Weiss, D. J., Su, S., Suen, K. Y., & Basford, J. (2022). Multidimensional computerized adaptive testing: A potential path toward the efficient and precise assessment of applied cognition, daily activity, and mobility for hospitalized patients. Archives of Physical Medicine and Rehabilitation, 103(5), S3-S14. https://doi.org/10.1016/j.apmr.2022.01.002
Weiss, D. J. (2013). Item banking, test development, and test delivery. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J.-I. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology, Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 185–200). American Psychological Association. https://doi.org/10.1037/14047-010