Developing and validating in-house, standardised, English reading and listening placement/streaming and achievement tests aligned to the Common European Framework of Reference Levels A2–B1 at a Japanese university

The research project presented in this thesis investigates one aspect of redesigning a compulsory English as a foreign language curriculum at a Japanese university to match intended target CEFR proficiency levels. Specifically, this thesis focuses on the creation and validation of institutional standardised tests of English reading and listening proficiency intended to be aligned to levels A2 and B1 of the CEFR. Kane’s argument-based approach to test validation, which firstly elaborates an Interpretation/Use Argument (IUA), and secondly uses a validity argument to evaluate evidence gathered to support the IUA, is used as a framework for the validation of the tests in question, known as the Bunkyo English Tests (BETs). Inferences in the IUA are also drawn from Chapelle, Enright, and Jamieson (2008), and the warrant for the final inference is drawn from Bachman and Palmer (2010). To substantiate the validity argument, data were collected from a variety of sources, including test results and test specifications, course outlines, surveys administered to teachers and students, interviews with teachers and university senior administrators, student course grades and assessment results, and student results from two other standardised tests, the Oxford Online Placement Test® and the Test of English for International Communication (TOEIC®). Results indicate that the BETs seem to have functioned sufficiently as course placement tests, but not as class streaming tests to divide classes within courses. Further analysis shows that the tests did not function effectively as achievement tests aligned to CEFR levels A2 and B1. This study demonstrates that Kane’s approach to test validation is viable for small-scale, in-house testing programs in the development phase, as it facilitated the selection of validity evidence, the analysis of which exposed areas of weakness in the tests and the test specifications, indicating clear avenues for improvement. Results also point to the need for further validation research on in-house tests which aim for CEFR alignment.