01whole.pdf (3.48 MB)
Download file

Assessing the assessments: using an argument-based validity framework to assess the validity and use of an English placement system in a foreign language context

Download (3.48 MB)
thesis
posted on 28.03.2022, 17:42 by Robert Charles Johnson
Use of a single, standardised instrument to make high-stakes decisions about testtakers is pervasive in higher education. Contrary to longstanding best practices encouraged by researchers, professional organizations, test publishers, and many accrediting bodies, however, few, if any such institutions have endeavoured to meaningfully validate the instrument(s) they use for their specific context and purposes. The current study attempted to address this void by developing and applying an argument-based validation framework for two widely adopted placement assessment methods - a standardised placement test (Accuplacer), and a locally developed and marked writing sample - utilised by a 2-year higher education institution in the Pacific. A hybrid of two validation structures - Kane's interpretive model and Bachman's assessment use argument - was implemented in order to assure a balanced focus on both test score interpretation and test utilization. Various types and sources of evidence informed the study, including instrument outcomes, student course results, institutional practices and policies, test publisher data, and the opinions of stakeholders gathered via focus group interview and questionnaires. Results are argued to provide insights regarding a number of current issues in the literature, including: i) debates regarding the relative strengths and weaknesses of standardised tests and locally developed and marked writing samples for informing placement decisions; ii) the value of locally conducted validation efforts to evaluate the performance and impact of an institution's chosen assessment instruments, and identifying opportunities for improvement; and iii) the need for further argument-based validation studies, particularly those which attend to both test score interpretation and the longneglected area of test utilization, to be carried out wherever assessments are used to make decisions which impact stakeholders.

History

Table of Contents

1. Introduction to the study -- 2. Literature review: recent advances in test validity and validation -- 3. Literature review: empirical investigations relevant to the current study -- 4. Materials and methods -- 5. Results -- 6. Summary, discussion and conclusion

Notes

Includes bibliographical references "Research carried out at the College of the Marshall Islands. Submitted December 20, 2011 to the Department of Linguistics, Division of Linguistics and Psychology, Macquarie University. This thesis is presented for the degree of Doctor of Philosophy in Applied

Awarding Institution

Macquarie University

Degree Type

Thesis PhD

Degree

PhD, Macquarie University, Faculty of Human Sciences, Department of Linguistics

Department, Centre or School

Department of Linguistics

Year of Award

2012

Principal Supervisor

Mehdi Riazi

Rights

Copyright disclaimer: http://www.copyright.mq.edu.au Copyright Robert Charles Johnson 2012.

Language

English

Extent

1 online resource (x, 194 pages) illustrations, charts, graphs

Former Identifiers

mq:30434 http://hdl.handle.net/1959.14/284597 2134447