Over recent decades, digital technologies have offered new ways to support learning and teaching. The use of technology to enhance learning is a complex process which in turn makes effective evaluation of the technology application in education a challenging endeavour. Reliable comparison of the use of technologies in education could potentially help stakeholders determine which approaches are effective across implementations, which in turn can inform the design of technology-enhanced curriculum and guidelines for the development and use of technology in education. According to research conducted in this thesis, there is a lack of field-specific instrument or approach that facilitates the holistic evaluation of the use of technology in education. The main purpose in this thesis is to address the question “How can we comprehensively evaluate the use of technology in education?”
This thesis by publication is comprised of six interrelated studies. Firstly, a systematic literature review of 365 empirical papers was conducted to investigate what researchers evaluated when using technology in education. Eight evaluation themes emerged from the systematic literature review: learning outcomes, affective elements, behaviours, design, technology elements, pedagogy, presence/community, and institutional environment (Paper 1). To make sure that the results of the first systematic review were robust and comprehensive, a tertiary systematic literature review was carried out to investigate the trends in learning technology evaluation (Paper 2). The results indicated that the outcomes of past literature reviews could be entirely encapsulated by the eight identified dimensions, providing a validation that the dimensions were broadly applicable. Based on criticisms that the evaluation of technology use in education is often not theoretically aligned, a systematic analysis of theory underpinning evaluation was conducted (Paper 3). This analysis revealed that the diverse theories used tended to encapsulate only some aspects of educational technology use, but once again, the eight dimensions were able to comprehensively represent the different aspects of theories that were used to evaluate of use of technology in education.
Next, items for a survey instrument were identified and validated by exploring what field experts think is important to focus upon when evaluating the use of technology in education (Paper 4). There was an alignment between what the specialists felt were important and the eight constructs that had typically been the focus of empirical studies. These eight dimensions were also validated in an analysis of informal student evaluations of a Coursera course - Excel Skills for Business Specialisation (Paper 5). The results indicated that, once again, all aspects of student informal evaluations were able to be encapsulated within the eight dimensions, and at the same time the paper helped to characterise the nature of the course that was used for the final validation study. In the final study, the eight-factor model was tested by surveying 1,352 participants undertaking a Coursera open online course (Paper 6). The ultimate 28-item “Comprehensive Evaluation of Use of Technology in Education” (CEUTIE) instrument was validated using Confirmatory Factor Analysis (CFA) and qualitative feedback from participants. The CEUTIE model provides researchers and other educational stakeholders with the means to perform consistent, holistic and robust evaluation and comparison of the use of technology for learning and teaching across diverse educational contexts.
Table of Contents
Chapter 1. Introduction - Technology use in education -- Chapter 2. How is the use of technology in education evaluated? A systematic review (Paper 1) -- Chapter 3. Evaluation of technology use in education - Findings from a critical analysis of systematic literature reviews (Paper 2) -- Chapter 4. The use and alignment of theory in educational technology research (Paper 3) -- Chapter 5. What should we evaluate when we use technology in education? (Paper 4) -- Chapter 6. Investigating the characteristics of MOOCs according to eight dimensions: A case study (Paper 5) -- Chapter 7. Comprehensive Evaluation of the use of Technology in Education – Validation with a Cohort of Global Open Online Learners -- Chapter 8. Conclusion -- References -- Appendices.Notes
Thesis by publicationAwarding Institution
Macquarie UniversityDegree Type
Thesis PhDDegree
Doctor of PhilosophyDepartment, Centre or School
Macquarie School of EducationYear of Award
2023Principal Supervisor
Matt BowerAdditional Supervisor 1
John De NobileAdditional Supervisor 2
Yvonne BreyerRights
Copyright: The Author
Copyright disclaimer: https://www.mq.edu.au/copyright-disclaimerLanguage
EnglishExtent
374 pages
