Assessing the assessments: using an argument-based validity framework to assess the validity and use of an English placement system in a foreign language context
thesisposted on 2022-03-28, 17:42 authored by Robert Charles Johnson
Use of a single, standardised instrument to make high-stakes decisions about testtakers is pervasive in higher education. Contrary to longstanding best practices encouraged by researchers, professional organizations, test publishers, and many accrediting bodies, however, few, if any such institutions have endeavoured to meaningfully validate the instrument(s) they use for their specific context and purposes. The current study attempted to address this void by developing and applying an argument-based validation framework for two widely adopted placement assessment methods - a standardised placement test (Accuplacer), and a locally developed and marked writing sample - utilised by a 2-year higher education institution in the Pacific. A hybrid of two validation structures - Kane's interpretive model and Bachman's assessment use argument - was implemented in order to assure a balanced focus on both test score interpretation and test utilization. Various types and sources of evidence informed the study, including instrument outcomes, student course results, institutional practices and policies, test publisher data, and the opinions of stakeholders gathered via focus group interview and questionnaires. Results are argued to provide insights regarding a number of current issues in the literature, including: i) debates regarding the relative strengths and weaknesses of standardised tests and locally developed and marked writing samples for informing placement decisions; ii) the value of locally conducted validation efforts to evaluate the performance and impact of an institution's chosen assessment instruments, and identifying opportunities for improvement; and iii) the need for further argument-based validation studies, particularly those which attend to both test score interpretation and the longneglected area of test utilization, to be carried out wherever assessments are used to make decisions which impact stakeholders.
Table of Contents1. Introduction to the study -- 2. Literature review: recent advances in test validity and validation -- 3. Literature review: empirical investigations relevant to the current study -- 4. Materials and methods -- 5. Results -- 6. Summary, discussion and conclusion
NotesIncludes bibliographical references "Research carried out at the College of the Marshall Islands. Submitted December 20, 2011 to the Department of Linguistics, Division of Linguistics and Psychology, Macquarie University. This thesis is presented for the degree of Doctor of Philosophy in Applied
Awarding InstitutionMacquarie University
Degree TypeThesis PhD
DegreePhD, Macquarie University, Faculty of Human Sciences, Department of Linguistics
Department, Centre or SchoolDepartment of Linguistics
Year of Award2012
Principal SupervisorMehdi Riazi
RightsCopyright disclaimer: http://www.copyright.mq.edu.au Copyright Robert Charles Johnson 2012.
Extent1 online resource (x, 194 pages) illustrations, charts, graphs
Former Identifiersmq:30434 http://hdl.handle.net/1959.14/284597 2134447
placementassessmentPsychometricsPsychological tests -- EvaluationPsychological tests -- InterpretationEducational evaluationstudent successhigher educationEducational evaluation -- Cross cultural studiesEducational tests and measurementsPsychological testsPsychometrics -- Case studiesPsychological tests -- Interpretation -- Case studieslanguage testing