Computerized adaptive testing (CAT), widely utilized in large-scale standardized tests because of benefits such as individuality and time independence (Brown, 1997) as well as accuracy, has further value in terms of validity even in small-scale in-house tests. This study describes the construction of an English for Specific Purposes (ESP) CAT system with an item bank of lexical items. The items are developed through the analysis of relevant corpora, specifically corpora of science magazines and academic engineering journals. After a few pretesting and equating using common item method, the number of items reached 180. A pilot test was administered to 63 Japanese university students majoring in engineering. One half of them answered 16 items which were selected adaptively so that the targeting probability of answering correct could be 0.5 (CAT1). The other half answered 25 items with the targeting probability of answering correct 0.8 (CAT2). Both CATs theoretically end with the same precision (S.E.=0.5 logit). M-UCAT (Kimura, Ohnishi & Nagaoka, 2011), which is a Rasch-based CAT module for Moodle, was used for the platform for this CAT administration. M-UCAT is developed based on a BASIC program of UCAT (Linacre, 1987). Ability estimation results for both groups were almost identical (CAT1: Mean=0.59, S.D.=.845; CAT2: Mean=0.62, S.D.=.841). At the end of all the tests but two, S.E. reached as low as 0.5 or less as expected. The actual percent correct for each CAT was lower than expected (CAT1: Mean=0.40, S.D.=.09; CAT2: Mean=0.65, S.D.=.06). This declination is mainly because the current item bank has a left-skewed distribution of item difficulty. It suggests that more items with middle and lower difficulty should be added to the item bank. The details of the pilot test, as well as the way to construct the item bank, are discussed to clarify its usefulness and shortcomings.
