Iteman: Running an Analysis

1. Running Item statistics using online exam results

This section describes how to run an analysis in Iteman. 

Iteman Data Import Process for Item Statistics in Item

To populate item statistics select from the Iteman Report, select the Auto Import item statistics for test.

The Item statistics will be updated for each of the items in the test:

For more information about how to prepare the input files, see Iteman: Preparing Input Files.  For more information on how to interpret output, see Iteman: The Scores Output File.

2. Importing Files

The first step in using Iteman is to specify the files you are uploading; these options are on the left side of the screen, and the list on the right side is the queue of past output files.

Skip/Omit Character: Use this field to specify specific characters in your data; this field would be used for an item that an examinee saw but did not leave an answer, for example.  It's important to differentiate from the Not Admin Character field.  

Not Admin Character: Use this field to specify specific characters in your data; this field would be used for an item was never seen by an examinee (e.g., unused items in a linear-on-the-fly test).

Output File Name: Type the file name you want to give your output file (e.g., 7Math-EOY). 

Report Title: Type the title that will be used inside the Word report. The report title is typically longer and more descriptive than the file name, as shown on the screen below. 

3. Scoring

The next step is to specify advanced scoring options.  Because Iteman is based on classical test theory, all examinees are scored with number-correct or sum scoring.  However, you have the option to convert that to scaled scoring or to classify examinees.

Scaled Scoring

This refers to the practice of converting scores to an arbitrary scale before reporting them.  There are many benefits to doings so.  Iteman supports the following two approaches: 

• You can convert the scale using a simple linear transformation. The raw scores are first multiplied by the slope coefficient and then the intercept is added to the product. For example, if you want the scores to be reported on a scale of 100 to 200 for a test of 50 items, the scaled score could be specified as SCALE = RAW × 2 + 100. 
• You can convert the standardized scores.  The raw scores are converted to have a mean of X and a standard deviation of Y. This form of scaling is useful if you desire to center the mean of the test around a constant value for use in a report. For example, the classic IQ scale with a mean=100 and SD=15.  Another common example of the latter is the T score, which converts the scale to a mean of 50 and standard deviation of 10.

Classify Examinees

You can classify examinees with pass/fail. Enter the cutscore you wish to use and use the option buttons to indicate if the score is a raw or scaled score. For example, if your test has 50 items and you want everyone with 30 or more to pass, click the Scaled score option button and enter 30 as the cutscore.

3. Output

The final step is to specify the options for the analysis and output.  Definitions for these follow.

Flagging Thresholds

Item statistic flagging allows you to specify an acceptable range for a statistic. For example, if you want to identify all items that have a P (proportion correct) below 0.40 (too hard) and above 0.98 (too easy), it can be specified here, and the output will label items with low P as “LP” and high P as “HP.” The “acceptable item mean” range is used to flag the item means of polytomous items; if your items are all "rate on a scale of 1 to 5" you might want to flag items with an average below 1.5 or above 4.5.  More information is in this blog post. 

Spuriousness

If you want to have the point-biserial and biserial correlations corrected for spuriousness, click the check box next in the Statistics Options area. Spuriousness refers to the fact that an item’s scores are included in the total score, so correlating an item with the total score implies that it is being correlated with itself to some extent. This effect is negligible if there are a large number of items on the test (e.g., more than 30); Iteman provides the option to correct for this issue, which should be utilized for tests of 30 items or less.

Skip Scoring

Selecting the Yes, consider Skip as incorrect option box defines skips/omits as having the full complement of option statistics computed for them. 

Decimal Places

This option simply formats your output ( e.g., do you want a P value of 0.748 or have it rounded to 0.75?). 

Groups for Quantiles

This drop-down list allows you to increase or decrease the number of examinee groups used for constructing the quantile plots. This number can range from 2 to 7. Larger numbers of points are recommended only for large sample sizes of at least 1,000 examinees.

Differential Item Functioning (DIF)

DIF is an analysis of bias; if the item is easier for an ethnic majority, using the Mantel-Haenszel approach, for example. This is limited to dichotomously scored items.

• This will use the data in the Groups column of your input.  Just specify which characters define the focal and reference groups.
• Next, specify the number of slices to be used for the calculation; this does not have to be the same as the quantile plots, but it often makes sense to do so.
  
  

4. Running the Report

When you are done with all the options, click Run.  The software displays your output file in the queue when it is ready.  You can then download and save and/or delete the file, if you wish.