Initialization Options for Adaptive Assessments

This algorithm uses the Rasch model and Item Response Theory to select Items or Activities with a psychometrically-calibrated difficulty. This difficulty needs to be set as part of the Items or Activities in the Item bank.

• In itemadaptive, Items are selected based on responses to previous Items.
• In branching, Activities (testlets) are selected instead of Items. The Activities themselves can be adaptive or fixed-form. The user's performance on the Activities as well as their Items will be used to calculate their ability estimate.

Either termination_criteria or sequence is mandatory.

This is a lightweight form of dynamic assessment, with conditional branches pre-configured in the Activity definition.

This option allows to delegate the Item selection to a third-party service implementing the Self-Hosted API.

type

required_tags

The Tags from which Items must be selected. Where multiple Tags of the same type are specified, the algorithm will alternate between each Tag.

By default, the Tags will be distributed equally, but the weighting can be customised by providing the desired weight in an object alongside the name.

Using the JSON example, all Items in the session would be tagged with subject:Math.

Half of the Items would be tagged with technology-enhanced-item:Yes, and the other half with technology-enhanced-item:No.

60% of the Items would be tagged with chapter:1 - The Number System, and 40% would be tagged with chapter:2 - Statistics and Probability.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.

initial_ability

The initial ability measure for the user. This will be used as the target difficulty for the first Item.

Default: 0

Example: 0

item_difficulty_tolerance

The size of the range of difficulties from which an Item should be randomly selected. For example, if the target difficulty is 0 logits, a tolerance of 1 will select an Item at random from all Items with a difficulty between -0.5 and 0.5 logits. If no Items are found in that range, an Item with difficulty between 0.5 and 1.5 will be selected, followed by the range -1.5 to -0.5, and so on.

Default: 1

Example: 0

item_difficulty_offset

An offset to apply to each difficulty target. A positive value will cause more difficult Items to be selected.

Default: 0

Example: -0.3

max_difficulty_change

When an ability estimate for the user is not yet available, this value is added or subtracted (depending on the user's response) from the difficulty of the previous Item to determine the difficulty target for the next Item.

Default: 0.7

Example: 0.7

max_item_difficulty

The maximum difficulty of all available Items.

Default: 9

Example: 9

min_item_difficulty

The minimum difficulty of all available Items.

Default: -9

Example: -9

interim_ability_estimation

Ordered list of the preferred estimation methods to use during the session, i.e. to determine the target difficulty for the next Item.

Default: [ "mle", "eap" ]

Example: [ "mle", "eap" ]

final_ability_estimation

Ordered list of the preferred estimation methods to use for the final ability estimate calculated at the end of the session.

Default: [ "mle", "score-lookup", "eap" ]

Example: [ "mle", "score-lookup", "eap" ]

subscore_ability_estimation

Ordered list of the preferred estimation methods to use for subscores.

Default: [ "empirical-eap", "mle" ]

Example: [ "empirical-eap", "mle" ]

eap

Information used to calculate eap ability estimates.

Example:

{
  "mean": -0.25,
  "num_points": 45,
  "standard_deviation": 0.95,
  "theta_max": 2.25,
  "theta_min": -3
}

empirical-eap

Information used to calculate empirical-eap ability estimates. These values override those in eap.

Example:

{
  "standard_deviation": 1
}

score-lookup

The lookup table used for score-lookup estimates. min and max are mapped to the lowest and highest possible scores, respectively.

Example:

{
  "min": -3.5,
  "max": 3.5
}

termination_criteria

The criteria used to determine when to finish a user's session (by order of precedence). Either error_below or max_items is mandatory.

{
  "error_below": 0.65,
  "max_items": 45,
  "min_items": 20
}

blacklist_tags

The Tags from which Items must not be selected.

Example:

{
  "unit": [
    "2.7 - Conditional statements"
  ]
}

seeding

Instructions for how seeding Items should be included in the session for calibration purposes. These Items are not expected to have difficulty measures, and responses to these Items have no effect on the user's ability estimate for the session.

tagtype and tagname define the specific Tag that identifies Items for seeding.

weight defines the proportion of seed Items that should be selected in the session -0.1 or 10% in this example.

Example:

{
  "tagtype": "lifecycle",
  "tagname": "seeding",
  "weight": 0.1,
  "earliest_seed_item": 3,
  "latest_seed_item": 3 // i.e. 3rd last
}

exclude_items

Items which should not be selected in the session. type can be hard for Items that should never be selected, or soft for Items that should only be selected if no similar Items are available.

Example:

{
  "type": "hard",
  "references": [
    "item-A",
    "item-B",
    "item-C"
  ]
}

sequence

Specific requirements for each individual Item in the session, allowing full control of the profile of every Item. The session will end when the end of this sequence is reached.

required_tags are the Tags from which the Item must be selected.

blacklist_tags are the Tags from which the Item must not be selected.

In this example, the first Item in the session must be from Unit 2.7, and must not be technology enhanced.

The second Item must be tagged with lifecycle:seeding. As this Tag matches the seeding Tag defined above, it will be treated as a seed Item, i.e, it will not be expected to have a difficulty, and responses to it will not be used when calculating ability estimates.

Example:

[
  {
      "required_tags": {
          "unit": [
              "2.7 - Conditional statements"
          ]
      },
      "blacklist_tags": {
          "technology-enhanced": [
              "Yes"
          ]
      }
  },
  {
      "required_tags": {
          "lifecycle": [
              "seeding"
          ]
      }
  }
]

min_items

Minimum number (> 1) of Items to present. The session will not finish until at least that many Items have been presented.

max_items

Maximum number (> 1) of Items to present. The session will finish as soon as that many Items have been presented.

error_below

Threshold on the standard error of measurement of the user's ability estimate. The session will finish when the estimate's standard error falls below this threshold. This value should be set in consultation with psychometricians calibrating the Items' difficulty.

type

steps

An Item branching assessment comprises an array of steps. Steps can define the reference of Items to present to the student when visited, or decisions to select which branch to visit next.

Only Item steps result in Items being presented to the student, decision steps are executed in the backend until the next Item step is visited.

The assessment starts at the first step defined in the array, which should be an Item step. The order of the rest of the array does not matter.

The most basic step provides an Item reference to be presented to the student when visited, as well as an unconditional pointer to the next step.

A basic step providing an Item reference, along with a conditional decision based on whether the student answered correctly or not.

Specify a global binary decision, depending on whether the total score is at least some ratio of the maximum achievable so far, or not.

id

The identifier of the step within the Activity.

reference

Reference of the Item to present at this step.

next

Either null (end of the assessment), the ID of the next step to unconditionally jump to.

id

The identifier of the step within the Activity.

reference

Reference of the Item to present at this step.

next

An object with the identifiers of the steps to jump to depending on whether the last answer to the Item was correct or not.

correct

Either null (end of the assessment), or the ID of the next step to jump to when the response to this Item is correct.

incorrect

Either null (end of the assessment), or the ID of the next step to jump to when the response to this Item is incorrect.

id

The identifier of the step within the Activity.

type

percentage

An integer between 0 and 100 representing the cut-off ration of the maximum score.

>=

Either null (end of the assessment), or the ID of the next step to jump to when the current score is greater than or equal to, the percentage of the maximum achievable score so far.

<

Either null (end of the assessment), or the ID of the next step to jump to when the current score is smaller than percentage of the maximum achievable score so far.

type

required_tags

The Tags from which Items must be selected.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.

mle

Maximum-Likelihood Estimation - this method requires a mixed response pattern for the Items seen, i.e., at least one correct response and one incorrect response.

eap

Expected A Posteriori (EAP) - this method requires eap to be configured.

empirical-eap

Empirical Expected A Posteriori (EAP) - as above, but the mean used for the calculation is the final ability estimate for the overall session. empirical_eap can optionally be configured.

score-lookup

Use the raw score for the session (the sum of correct Items) to select an ability estimate from a lookup table - this method requires score_lookup to be configured (see below).