01whole.pdf (3.48 MB)
Assessing the assessments: using an argument-based validity framework to assess the validity and use of an English placement system in a foreign language context
thesis
posted on 2022-03-28, 17:42 authored by Robert Charles JohnsonUse of a single, standardised instrument to make high-stakes decisions about testtakers is pervasive in higher education. Contrary to longstanding best practices encouraged by researchers, professional organizations, test publishers, and many accrediting bodies, however, few, if any such institutions have endeavoured to meaningfully validate the instrument(s) they use for their specific context and purposes. The current study attempted to address this void by developing and applying an argument-based validation framework for two widely adopted placement assessment methods - a standardised placement test (Accuplacer), and a locally developed and marked writing sample - utilised by a 2-year higher education institution in the Pacific. A hybrid of two validation structures - Kane's interpretive model and Bachman's assessment use argument - was implemented in order to assure a balanced focus on both test score interpretation and test utilization. Various types and sources of evidence informed the study, including instrument outcomes, student course results, institutional practices and policies, test publisher data, and the opinions of stakeholders gathered via focus group interview and questionnaires. Results are argued to provide insights regarding a number of current issues in the literature, including: i) debates regarding the relative strengths and weaknesses of standardised tests and locally developed and marked writing samples for informing placement decisions; ii) the value of locally conducted validation efforts to evaluate the performance and impact of an institution's chosen assessment instruments, and identifying opportunities for improvement; and iii) the need for further argument-based validation studies, particularly those which attend to both test score interpretation and the longneglected area of test utilization, to be carried out wherever assessments are used to make decisions which impact stakeholders.
History
Table of Contents
1. Introduction to the study -- 2. Literature review: recent advances in test validity and validation -- 3. Literature review: empirical investigations relevant to the current study -- 4. Materials and methods -- 5. Results -- 6. Summary, discussion and conclusionNotes
Includes bibliographical references "Research carried out at the College of the Marshall Islands. Submitted December 20, 2011 to the Department of Linguistics, Division of Linguistics and Psychology, Macquarie University. This thesis is presented for the degree of Doctor of Philosophy in AppliedAwarding Institution
Macquarie UniversityDegree Type
Thesis PhDDegree
PhD, Macquarie University, Faculty of Human Sciences, Department of LinguisticsDepartment, Centre or School
Department of LinguisticsYear of Award
2012Principal Supervisor
Mehdi RiaziRights
Copyright disclaimer: http://www.copyright.mq.edu.au Copyright Robert Charles Johnson 2012.Language
EnglishExtent
1 online resource (x, 194 pages) illustrations, charts, graphsFormer Identifiers
mq:30434 http://hdl.handle.net/1959.14/284597 2134447Usage metrics
Categories
Keywords
placementassessmentPsychometricsPsychological tests -- EvaluationPsychological tests -- InterpretationEducational evaluationstudent successhigher educationEducational evaluation -- Cross cultural studiesEducational tests and measurementsPsychological testsPsychometrics -- Case studiesPsychological tests -- Interpretation -- Case studieslanguage testing