Computerized Adaptive Testing (CAT)

Computerized adaptive testing (CAT) is a dynamic paradigm for assessment.  Instead of delivering the same exam to every examinee, the exam is personalized for every student.  This includes adapting the difficulty of the test (which items they see) and the duration of the test (how many items they see).  Examinees are scored with item response theory (IRT).  Learn more about the methods at this blog post.

Implementing a CAT in Assess.ai

In Assess.ai, CAT is defined at a Test level.  Follow these steps to implement.

1. Create a new Test.  Specify the Test Type as Adaptive.  This can be done during the creation screen, or afterwards on the About screen as seen below.

Define the Content Constraints for the Test

NOTE: Items need to be tagged when using the Content Constraints feature.  For more information about Tags and Tagging click here: Tags

Select Add Element

Select the Tag Group e.g. English and the Tag e.g. Grammar

Then select the percentage (%) of English Grammar Items to include in the Test.

Continue adding Elements required from the English Tag Group

NOTE: The Total must be 100%

Then click on Save

The Test will include items based on the structure selected in the Content Constraints.

Set the Adaptive Options for the Test

Specify the min/max theta range and the target SEM for the CAT, using the screen shown above.

The example above:

Min Theta = -3

Max Theta = 3

Termination Criterion

Target SEM (Structural Equation Modelling) = 0,4

Minimum Items: = 15 (each student will receive a minimum of 15 questions)  

Maximum Items: = 20 (each student will receive a maximum of 20 questions)

Note: To set an adaptive test as Fixed Length, set the Minimum and Maximum items to the same number of items. e.g. Minimum items = 15, Maximum items = 15

 

2. Set the scoring method to item response theory (IRT).  Specify the min/max theta range and the target SEM for the CAT, using the screen shown above.

3. Add items to the Section for the test (only one section is allowed). 

Note: Make sure that all items have dichotomous IRT parameters; this should be done during item banking on the Statistics Screen.

See Item Statistics screen below:

Items that do not have the dichotomous IRT parameters recorded and are included in a CAT test will block a CAT Test from being locked.

Setting up the Session

The Session setup is the same as a linear test:

Note: The Session Type is Standard (independent tests)

Enemy items:

Enemy Items are setup by going to the Item and selecting the About Tab.

Select any Enemy Item to the Item being edited.

In addition, the reciprocal item will be flagged.

Example: Item ei - 001 has an enemy item ei - 002 therefore item ei- 002 will automatically be flagged with enemy item ei - 001

 

If you have questions about this process, please contact our Support team.