Initialization Options for Adaptive Assessments

ItemTestletAdaptive | ItemBranching | SelfHosted
When present, this object configures the Items API to present Items dynamically, rather than using a hardcoded list. The next Item(s) is/are selected based on the user's performance so far. Multiple Item selection algorithms can be selected and configured through this object.

See Adaptive and Dynamic Assessments for more information.

Types
ItemTestletAdaptive

This algorithm uses the Rasch model and Item Response Theory to select Items or Activities with a psychometrically-calibrated difficulty. This difficulty needs to be set as part of the Items or Activities in the Item bank.

  • In itemadaptive, Items are selected based on responses to previous Items.
  • In branching, Activities (testlets) are selected instead of Items. The Activities themselves can be adaptive or fixed-form. The user's performance on the Activities as well as their Items will be used to calculate their ability estimate.

Either termination_criteria or sequence is mandatory.

ItemBranching
This is a lightweight form of dynamic assessment, with conditional branches pre-configured in the Activity definition.
SelfHosted
This option allows to delegate the Item selection to a third-party service implementing the Self-Hosted REST API.

See Adaptive and Dynamic Assessments for more information.

object

This algorithm uses the Rasch model and Item Response Theory to select Items or Activities with a psychometrically-calibrated difficulty. This difficulty needs to be set as part of the Items or Activities in the Item bank.

  • In itemadaptive, Items are selected based on responses to previous Items.
  • In branching, Activities (testlets) are selected instead of Items. The Activities themselves can be adaptive or fixed-form. The user's performance on the Activities as well as their Items will be used to calculate their ability estimate.

Either termination_criteria or sequence is mandatory.

See Item or Testlet Adaptive for more information.

Properties:
type
Type: string
Possible values:
  • itemadaptive
  • branching
required_tags

The Tags from which Items must be selected. Where multiple Tags of the same type are specified, the algorithm will alternate between each Tag.

By default, the Tags will be distributed equally, but the weighting can be customised by providing the desired weight in an object alongside the name.

Using the JSON example, all Items in the session would be tagged with subject:Math.

Half of the Items would be tagged with technology-enhanced-item:Yes, and the other half with technology-enhanced-item:No.

60% of the Items would be tagged with chapter:1 - The Number System, and 40% would be tagged with chapter:2 - Statistics and Probability.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.
initial_ability
Type: float (logit)
The initial ability measure for the user. This will be used as the target difficulty for the first Item.
Default: 0
Example: 0

See logits for more information.

item_difficulty_tolerance
Type: float (logit)
The size of the range of difficulties from which an Item should be randomly selected. For example, if the target difficulty is 0 logits, a tolerance of 1 will select an Item at random from all Items with a difficulty between -0.5 and 0.5 logits. If no Items are found in that range, an Item with difficulty between 0.5 and 1.5 will be selected, followed by the range -1.5 to -0.5, and so on.
Default: 1
Example: 0

See logits for more information.

item_difficulty_offset
Type: float (logit)
An offset to apply to each difficulty target. A positive value will cause more difficult Items to be selected.
Default: 0
Example: -0.3

See logits for more information.

max_difficulty_change
Type: float (logit)
When an ability estimate for the user is not yet available, this value is added or subtracted (depending on the user's response) from the difficulty of the previous Item to determine the difficulty target for the next Item.
Default: 0.7
Example: 0.7

See logits for more information.

max_item_difficulty
Type: float (logit)
The maximum difficulty of all available Items.
Default: 9
Example: 9

See logits for more information.

min_item_difficulty
Type: float (logit)
The minimum difficulty of all available Items.
Default: -9
Example: -9

See logits for more information.

interim_ability_estimation
Ordered list of the preferred estimation methods to use during the session, i.e. to determine the target difficulty for the next Item.
Default: [ "mle", "eap" ]
Example: [ "mle", "eap" ]
final_ability_estimation
Ordered list of the preferred estimation methods to use for the final ability estimate calculated at the end of the session.
Default: [ "mle", "score-lookup", "eap" ]
Example: [ "mle", "score-lookup", "eap" ]
subscore_ability_estimation
Ordered list of the preferred estimation methods to use for subscores.
Default: [ "empirical-eap", "mle" ]
Example: [ "empirical-eap", "mle" ]
eap
Type: object
Information used to calculate eap ability estimates.
Example:
{
  "mean": -0.25,
  "num_points": 45,
  "standard_deviation": 0.95,
  "theta_max": 2.25,
  "theta_min": -3
}
empirical-eap
Type: object
Information used to calculate empirical-eap ability estimates. These values override those in eap.
Example:
{
  "standard_deviation": 1
}
score-lookup
Type: object
The lookup table used for score-lookup estimates. min and max are mapped to the lowest and highest possible scores, respectively.
Example:
{
  "min": -3.5,
  "max": 3.5
}
termination_criteria
Type: object
The criteria used to determine when to finish a user's session (by order of precedence). Either error_below or max_items is mandatory.
blacklist_tags
Type: object
The Tags from which Items must not be selected.
Example:
{
  "unit": [
    "2.7 - Conditional statements"
  ]
}
seeding
Type: object

Instructions for how seeding Items should be included in the session for calibration purposes. These Items are not expected to have difficulty measures, and responses to these Items have no effect on the user's ability estimate for the session.

tagtype and tagname define the specific Tag that identifies Items for seeding.

weight defines the proportion of seed Items that should be selected in the session -0.1 or 10% in this example.

Example:
{
  "tagtype": "lifecycle",
  "tagname": "seeding",
  "weight": 0.1,
  "earliest_seed_item": 3,
  "latest_seed_item": 3 // i.e. 3rd last
}
exclude_items
Type: object
Items which should not be selected in the session. type can be hard for Items that should never be selected, or soft for Items that should only be selected if no similar Items are available.
Example:
{
  "type": "hard",
  "references": [
    "item-A",
    "item-B",
    "item-C"
  ]
}
sequence
Type: array

Specific requirements for each individual Item in the session, allowing full control of the profile of every Item. The session will end when the end of this sequence is reached.

required_tags are the Tags from which the Item must be selected.

blacklist_tags are the Tags from which the Item must not be selected.

In this example, the first Item in the session must be from Unit 2.7, and must not be technology enhanced.

The second Item must be tagged with lifecycle:seeding. As this Tag matches the seeding Tag defined above, it will be treated as a seed Item, i.e, it will not be expected to have a difficulty, and responses to it will not be used when calculating ability estimates.

Example:
[
  {
      "required_tags": {
          "unit": [
              "2.7 - Conditional statements"
          ]
      },
      "blacklist_tags": {
          "technology-enhanced": [
              "Yes"
          ]
      }
  },
  {
      "required_tags": {
          "lifecycle": [
              "seeding"
          ]
      }
  }
]
object
The criteria used to determine when to finish a user's session (by order of precedence). Either error_below or max_items is mandatory.
Properties:
min_items
Type: integer
Minimum number (> 1) of Items to present. The session will not finish until at least that many Items have been presented.
max_items
Type: integer
Maximum number (> 1) of Items to present. The session will finish as soon as that many Items have been presented.
error_below
Type: integer
Threshold on the standard error of measurement of the user's ability estimate. The session will finish when the estimate's standard error falls below this threshold. This value should be set in consultation with psychometricians calibrating the Items' difficulty.
{
  "error_below": 0.65,
  "max_items": 45,
  "min_items": 20
}
object
This is a lightweight form of dynamic assessment, with conditional branches pre-configured in the Activity definition.

See Building Item Branching Assessments for more information.

Properties:
type
Type: string
Possible value(s): itembranching
required_tags

The Tags from which Items must be selected.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.
steps

An Item branching assessment comprises an array of steps. Steps can define the reference of Items to present to the student when visited, or decisions to select which branch to visit next.

Only Item steps result in Items being presented to the student, decision steps are executed in the backend until the next Item step is visited.

The assessment starts at the first step defined in the array, which should be an Item step. The order of the rest of the array does not matter.

An Item branching assessment comprises an array of steps. Steps can define the reference of Items to present to the student when visited, or decisions to select which branch to visit next.

Only Item steps result in Items being presented to the student, decision steps are executed in the backend until the next Item step is visited.

The assessment starts at the first step defined in the array, which should be an Item step. The order of the rest of the array does not matter.

Types
ItemBranchingStepItem
The most basic step provides an Item reference to be presented to the student when visited, as well as an unconditional pointer to the next step.
ItemBranchingStepBinary
A basic step providing an Item reference, along with a conditional decision based on whether the student answered correctly or not.
ItemBranchingStepGlobalScore
Specify a global binary decision, depending on whether the total score is at least some ratio of the maximum achievable so far, or not.
object
The most basic step provides an Item reference to be presented to the student when visited, as well as an unconditional pointer to the next step.
Properties:
id
Type: string
The identifier of the step within the Activity.
reference
Type: string
Reference of the Item to present at this step.
next
Type: string|null
Either null (end of the assessment), the ID of the next step to unconditionally jump to.
{
  "id": "step-1",
  "reference": "item-reference-1",
  "next": "step-2"
}
object
A basic step providing an Item reference, along with a conditional decision based on whether the student answered correctly or not.
Properties:
id
Type: string
The identifier of the step within the Activity.
reference
Type: string
Reference of the Item to present at this step.
next
Type: object
An object with the identifiers of the steps to jump to depending on whether the last answer to the Item was correct or not.
{
  "id": "step-2",
  "reference": "item-reference-2",
  "next": {
    "correct": "global-decision-step-1",
    "incorrect": "step-3"
  }
}
An object with the identifiers of the steps to jump to depending on whether the last answer to the Item was correct or not.
Properties:
correct
Type: string|null
Either null (end of the assessment), or the ID of the next step to jump to when the response to this Item is correct.
incorrect
Type: string|null
Either null (end of the assessment), or the ID of the next step to jump to when the response to this Item is incorrect.
object
Specify a global binary decision, depending on whether the total score is at least some ratio of the maximum achievable so far, or not.
Properties:
id
Type: string
The identifier of the step within the Activity.
type
Type: string
Possible value(s): global-score
percentage
Type: integer
An integer between 0 and 100 representing the cut-off ration of the maximum score.
>=
Type: string|null
Either null (end of the assessment), or the ID of the next step to jump to when the current score is greater than or equal to, the percentage of the maximum achievable score so far.
<
Type: string|null
Either null (end of the assessment), or the ID of the next step to jump to when the current score is smaller than percentage of the maximum achievable score so far.
{
  "<": "step-4",
  ">=": null,
  "id": "global-decision-step-1",
  "percentage": 50,
  "type": "global-score"
}
object
This option allows to delegate the Item selection to a third-party service implementing the Self-Hosted REST API.

See Full object specification for more information.

Properties:
type
Type: string
Possible value(s): selfhosted
required_tags

The Tags from which Items must be selected.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.
string
Method to use for calculating ability estimates.
Properties:
mle
Type: string
Maximum-Likelihood Estimation - this method requires a mixed response pattern for the Items seen, i.e., at least one correct response and one incorrect response.
eap
Type: string
Expected A Posteriori (EAP) - this method requires eap to be configured.
empirical-eap
Type: string
Empirical Expected A Posteriori (EAP) - as above, but the mean used for the calculation is the final ability estimate for the overall session. empirical_eap can optionally be configured.
score-lookup
Type: string
Use the raw score for the session (the sum of correct Items) to select an ability estimate from a lookup table - this method requires score_lookup to be configured (see below).
object

The Tags from which Items must be selected. Where multiple Tags of the same type are specified, the algorithm will alternate between each Tag.

By default, the Tags will be distributed equally, but the weighting can be customised by providing the desired weight in an object alongside the name.

Using the JSON example, all Items in the session would be tagged with subject:Math.

Half of the Items would be tagged with technology-enhanced-item:Yes, and the other half with technology-enhanced-item:No.

60% of the Items would be tagged with chapter:1 - The Number System, and 40% would be tagged with chapter:2 - Statistics and Probability.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.
{
    "chapter": [
        {"name": "1 - The Number System", "weight": 3},
        {"name": "2 - Statistics and Probability", "weight": 2}
    ],
    "subject": ["Math"],
    "technology-enhanced-item": ["Yes", "No"]
}
array

The Tags from which Items must be selected.

Note: This is also used to select Items for automatic inclusion in Item pools alongside the Activity.
[
  { "type": "subject", "name": "Math" },
  { "type": "technology-enhanced-item", "name": "yes" },
  { "type": "technology-enhanced-item", "name": "no" },
  { "type": "chapter", "name": "1 - The Number System", "weight": 3 },
  { "type": "chapter", "name": "2 - Statistics and Probability", "weight": 2 }
]