MultiStage Testing is a dynamic approach to assessment that blends aspects of adaptive testing and traditional (linear) testing. Instead of adapting the difficulty of the test after every item, we pre-define blocks of items (called Testlets, Panels, or Modules), and adapt after each Testlet.
Consider the following example, which has 3 stages, using a 1-3-3 design.
- Stage 1: A vocabulary test of 15 items
- Stage 2: A nonfiction reading passage with 5 associated items (3 possible passages: Low, Medium, High difficulty)
- Stage 3: A fiction reading passage with 5 associated items (3 possible passages: Low, Medium, High difficulty)
The purpose of MST, in the example above, is to deliver an appropriate level reading passage to each examinee.
i.e. If a "high" difficulty level passage is delivered to an entry level student or vice versa, the student will be lost.
The traditional approach of giving all examinees a middle difficulty passage is not ideal either, since it does a disservice to both the top and lower level students.
How to implement multistage testing in Assess.ai
Assess.ai deals with the complex construction by treating each Stage as a Test, and each possible Testlet / Panel / Module as a Section. The second (and, if needed, third) stages must have a Test that is specifically "Testlet-based." The initial test, which is called a Routing Test in the literature, is a standard Linear Test in Assess.ai.
Here are the steps to implement a MultiStage Test.
Creating a Test
Create a new test, and set the Test Type to be "Testlet-based."
You can do this on the initial Test creation screen, or edit the About screen of the test as seen below.
All Tests in a MultiStage Test must be scored with Item Response Theory (IRT) using the same parameters.
To edit the Scoring Properties of a test:
- Select the Test
- On the Edit Screen Select Scoring
- Scoring method = IRT score (theta)
- Select Maximum Likelihood
2. Create a Section for each of the intended Testlets.
Note: To edit the name / description of a Section click on the Cog Icon (It helps to name the sections appropriately).
As the Sections (Stages) are added to a MultiStage test the Infographic will populate with Information about the items in each Section (Stage). (Future Release)
- Stage 2 Easy
- Stage 2 Medium
- Stage 2 Hard
Creating a MultiStage Session
3. Create a Session that is a MultiStage Session Type.
3a. Include the necessary tests. An example is shown below.
Note: Tests must be locked before being added to a Session
The Stage 2 and Stage 3 tests have both been constructed as Testlet-Based and have three sections. (Future Release)
Stage 2: Stage 3:
There is no need to define any calculations for selecting the appropriate Section/Testlet; this will be done automatically using the same approach used in most adaptive tests: selecting the one with the most Information at the current theta estimate.
During Multi Stage Test Delivery the candidate will not be able to skip a question.
The following message will display if a candidate attempts to skip a question:
Unscored Items can be included in Multistage Tests.
If you have questions about this process, please contact our Support team.