Abstract:

Many international large-scale educational survey studies, such as TIMSS, PIRLS and PISA, adopted the incomplete balanced booklet design to make sure that the estimated ability of examinees does not covariate with the position of item blocks. By this incomplete design every two booklets contain some common items, and the ability scales of booklets drawn from the item response theory can be equated across booklets through these common items. However, the positional effect of item blocks exists somehow within a booklet, which helps in understanding the influential factors of the item difficulty. For TIMSS 2007, a student spends two class periods to complete a booklet, and each class period consists of two item blocks. The current study thus explored the effects of the class period (first vs. second class period), the position of item block within a class period (first vs. second position), and the country (Taiwan vs. USA) on the item difficulty of TIMSS 2007 by treating them as facets of item difficulty in the generalized Rasch model. As the interaction suggests a practice effect in the first class period where an item was less difficult at the second position than the first position, and a fatigue effect in the second class period where an item was more difficult at the second position than the first position across countries, the USA demonstrated a more serious fatigue effect than Taiwan. Consequently, the serious fatigue effect caused the items more difficult and lowered the ability estimate of examinees of the USA in the end of testing.

Keywords:

Generalized Rasch Model, IRT, large-scale assessment, TIMSS.