Creating a Test

Creating a New Test

Select Tests from the Main Menu

To create a new Test click on the Add Test button:

The Add new Test window will display, enter a unique Test Name and Description.

Select the Test Type:

Linear	Test items / questions are delivered in the same order for each examinee.
Linear-on-the-Fly Testing (LOFT)	Different items are delivered to each examinee from a pool of items defined by content constraints in the test setup. Each examinee receives the same number of items.
Adaptive (CAT)	Test adapts to the performance of the  Candidate.
Testlet Based (MST)	Similar to CAT, however, items are selected in groups of items (Testlet) rather than by individual Items.

then click on Add new.  

Note: there are three tabs at the top of the screen - Assembly, About and Scoring. 

• The Assembly tab is where the items for the Test are set up.  
• The About tab stores key information about the test (e.g. Test Name, Description, Time Limits and Tags).
• The Scoring tab is where the scoring requirements for the test are setup.

The Assembly Tab

The Assembly tab has a substantial amount of important functionality; its purpose is to allow you to assemble psychometrically sound test forms by finding appropriate items in your item banks. 

New Tests will be created with a default : First test section, this can be renamed and edited by clicking on the cog icon. 

Import Item Statistics	Import the items statistics for this test
Export Item Statistics	Export the item statistics for this test
Export Test to HTML	Export the test to HTML format
Export Metadata	Export the item metadata for this test
Export Test to Word	Export the Test to Word (see  for Printing Options)
Export Items to Word	Export the items to Word for this test

 

Test Sections

Enter the Section Name and Description.

• Check the Shuffle order of items check box to shuffle the order of items in the section.
• Enter the Time Limit for the Section.
• For individual item time limits the Section Check the : * Utilize item time limits check box. This will allow the user to specify a time limit per item (see note below about setting item time limits).
• Check the Display message to examinee that they are entering a timed section check box to enable this examinee delivery message.
• Check the Linear on the fly (LOFT) section check box for the section to be a LOFT delivery.
  • Select the Section Template.  (See notes on Section Templates: Automated Test Assembly)
• Pre section Instructions
  • Show pre section instructions.  This option allows you to enter the instructions for the section before delivering the first Item of the Section.
• Enter the instructions for the section in the text box provided.

The Examinee Screen with Pre Section instructions:

* Note: Setting the item time limits is done during the Test Assembly, after adding Items to the section:

• Delete unused Sections by clicking on the delete icon . Note: This will delete the section as well as all items contained in that section.

To Add Items to the Section click on the plus icon .

On saving the test, a check will be done for any duplicated section names.

Enemy items:

Enemy Items are setup by going to the Item and selecting the About Tab.  Add the items that are enemies to the item.

When adding items to a test, if an enemy item is selected an icon will display next to any corresponding enemy items, with the message - "this is an enemy item.

If enemy items are added to a test then the following messaging will display:

Enemy items will need to be removed from the test before saving is successful.

Finding and Selecting Items

There are several search tools available to find the items for a test.

Search: type in the name or part of the name of any item.

Filters: Item Banks and Tags

• Item Banks: The default setting is All Items Banks, use the drop-down arrow to select an Item Bank to search (e.g. Math Item Bank or Biology Item Bank). 
• Tags:  narrow the search by selecting a Tag within the Item Bank previously selected, (e.g. Item Bank = Math and Tag = Algebra). 
• Click on Apply after making search selections.

The list of items that match the filter criteria will be displayed (bottom left). Click on the eye icon to preview the details of an item . 

There are 2 ways to add items to the test:

• Click, drag and drop an item from the left panel over to the right panel.
• Select the items by ticking the check boxes next to each item and click on the Add Items button

             

• Click on the Add All (#) button to add all the items listed on the left to the test.

• The Remove Items button can be used to remove selected items from the test. 

To rearrange the sequence of items in the test, drag and drop the items in the right side panel.

To Scramble Items in a section select Scramble Items:

Anchor Items

Anchor items are items which overlap between test forms.

An anchor item is a common item administered in two or more alternative test forms with the aim of establishing the equivalence of the test scores on the alternative forms.

Anchor Items are set during the Assembly of a test.  When the test is duplicated the Anchor Items will be duplicated.

Note: Anchor Items are Scored Items.   If an item is unscored the Anchor Flag cannot be set.

Import Items 

The Import Items function can be used for a quick setup of a test.

Step 1. Export the Items from Item Bank

a. Go to the Item bank

b. Select the Items for the Test

c. Select Export Items

d. Enter a Name for the Exported Items file and select the File Type = Excel

e. Go to the Notifications and Download the Items Spreadsheet

Step 2. Import the Items to the Test Section

a. Go to the Test Builder

b. Select the Test Section and add Items to the Section

c. Select Import

d. Select the spreadsheet previously exported from Items

e. The Test Section will populate with the items from the Spreadsheet.

f. Continue adding items to the section or Save & Exit.

The About Tab

The About tab stores key information about the test (e.g. Test Name, Description, Time Limits and Tags).

Test ID:	Short Name for the Test
Status:	The status of the Test (e.g. Active)  Note: The Status options are: New, Review, Active, Retired * Tests with a Retired Status are not available in Sessions.
Time Limit (in minutes)	Enter the time limit for the test.  This time is set in minutes, e.g. 60 minutes.  Note: Test Sections can also have a time limit, the sum of the sections cannot exceed the Total Test Time.
Enable Correct Answer Preview	This option is used by Test Administrators or Test Reviewers. When enabled this will allow the candidate to check their answers during test delivery.
Shuffle order of Items	Select this option to Shuffle to delivery order of the Items.
Test Type:	Linear - all test questions in the same order to all candidates Linear-on-the-Fly (LOFT) - Setup this type of test using Section Templates see notes: Automated Test Assembly Adaptive - CAT (Computer Adaptive Test) see notes: Computerized Adaptive Testing (CAT)  Testlet Based - MultiStage Tests (MST) - MST Set up Notes
Add a Calculator to the toolbar for all items:	This option turns on or off the calculator in the toolbar for the test.  If you want to have a calculator on specific items, add it as a Feature to that item.
Show a running count up clock to the Examinee Test Timer:	A count up clock can be inserted onto the Candidate Test Screen.  Default for this option is set to Off.
Description:	Enter a Test Description.
Add Tag:	Enter a tag(s) for the test.  Tags make searching easier.

 

The Scoring Tab

The Scoring Tab is where the scoring specifications for the test are setup.

Sub-scores - Automatically create Sub Scores using tags

Items for the Test:
Ensure that all items are tagged by group for Test Sub-scoring.
Example Tag Group:
• English
    o English-Reading
    o English-Writing
    o English-Spelling

• A test may contain:
    • # items tagged with - English and English-Reading
    • # items tagged with - English and English-Writing
    • # items tagged with - English and English-Spelling

The resulting Score/Metadata report would have the scores for the English as well as sub scores for each of the Sub-groups (English-Reading , English-Writing, English-Spelling)

Scoring Methods

There are 3 scoring methods available:

• Sum of Points - Score as a Raw total
• Percentage - Score as a percentage
• IRT score (theta) - Item Response Theory Score - used for Adaptive Tests (CAT and Multistage)

Cutscore Options:

There are 5 methods for calculating the Cutscore of a test:

• Disable Pass/Fail Cutscore - No cutscore for the test
• Pass / Fail based on a Session Cutscore: Cutscore is set for the session .

Notes:

• There can only be one test is a session when using the Session CutScore
• A Cutscore cannot be set on both the Test and the Session
• The session cutscore feature is enabled by going to Admin / Configuration and selecting :Session
• Pass/Fail Cutscore based on overall Score (all sections):  Calculate the cutscore based on a minimum Sum of Points:
  • e.g. 39 out of 50 - Examinee must achieve a minimum of 39 out of a possible 50.

• The max score indicates the total score of the test.
• or Percentage - Calculate the cutscore based on a minimum Percentage
  • e.g. 78% - Examinee must achieve a minimum of 78% on the test.

• Pass/Fail Cutscore based on multiple hurdles (raw score for each section) 
  • Multiple Hurdle Scoring.  Sections have cut scores, if a candidate does not achieve in any one of the sections, regardless of overall score, the candidate will “fail” the test.

Scaled Scoring:

Scaled Scoring will alter the raw scores into an arbitrary scale, this is for reporting purposes.

Scaled scores are computed using the scaling function (detailed below) for the total number correct scores and/or the domain number-correct scores.  Scaled scoring is often used to mask details about the test, such as exact number of items or raw cutoff score, or to express scores on a different scale than number correct.  Most large-scale tests use a form of scaled scoring.

• Linear Transformation: The raw scores are first multiplied by the slope coefficient then the intercept is added to the product. 
  • For example: if you want the scores to be reported on a scale of 100 to 200 for a test of 50 items, the scaled score could be specified as SCALE = RAW × 2 + 100.
• Standardized Transformation: The raw scores are converted to have a mean of X and a standard deviation of Y.  This form of scaling is useful to center the mean of the test around a constant value for use in a report. 
  • For example: the classic IQ scale with mean=100 and SD=15.

Click the Save & Exit button to save and exit the test. 

The History Tab

The History Tab displays data about changes to the test.  This an audit trail per test. 

When an UPDATE event is selected, the details of the data that was changed will be displayed.