Assessing the assessments: using an argument-based validity framework to assess the validity and use of an English placement system in a foreign language context

Use of a single, standardised instrument to make high-stakes decisions about testtakers is pervasive in higher education. Contrary to longstanding best practices encouraged by researchers, professional organizations, test publishers, and many accrediting bodies, however, few, if any such institutions have endeavoured to meaningfully validate the instrument(s) they use for their specific context and purposes. The current study attempted to address this void by developing and applying an argument-based validation framework for two widely adopted placement assessment methods - a standardised placement test (Accuplacer), and a locally developed and marked writing sample - utilised by a 2-year higher education institution in the Pacific. A hybrid of two validation structures - Kane's interpretive model and Bachman's assessment use argument - was implemented in order to assure a balanced focus on both test score interpretation and test utilization. Various types and sources of evidence informed the study, including instrument outcomes, student course results, institutional practices and policies, test publisher data, and the opinions of stakeholders gathered via focus group interview and questionnaires. Results are argued to provide insights regarding a number of current issues in the literature, including: i) debates regarding the relative strengths and weaknesses of standardised tests and locally developed and marked writing samples for informing placement decisions; ii) the value of locally conducted validation efforts to evaluate the performance and impact of an institution's chosen assessment instruments, and identifying opportunities for improvement; and iii) the need for further argument-based validation studies, particularly those which attend to both test score interpretation and the longneglected area of test utilization, to be carried out wherever assessments are used to make decisions which impact stakeholders.