Skip to main content

Open Computer-based Assessment with the CBA ItemBuilder

5 Scoring of Tasks

This chapter describes how scoring can be defined in CBA ItemBuilder Projects Files. Scoring is always defined on the task level (see section 3.6 for details on defining tasks). Accordingly, before scoring can be implemented in newly created CBA ItemBuilder projects files, a Task must be configured (see section 3.6 for details). A second prerequisite for defining the scoring of Tasks is to define names for all required components. That means that User Defined Ids (see subsection 3.7.4 for details) is required for the components used to gather responses. Human readable User Defined Ids are suggested to remember the meaning of particular identifiers when using them, for instance, in the syntax definition of scoring rules. Since this definition of scoring rules is based on the User Defined Ids, systematically defined and easily readable IDs simplify the creation and validation of scoring rules for item authors.

The definition of explicit scoring rules is not mandatory for the use of CBA ItemBuilder items, but it provides the most flexible way to combine evidence into scoring. Alternatively, 1) the so-called component state, that is, the values selected or entered for all input elements, can be automatically stored by the test assembly and deployment software (see chapter 7). Moreover, 2) the components can be linked to FSM variables (applies to version 10.4) and the last value of FSM variables when a task is exited can be used. Finally, 3), log data can also contain all changes of component values and it is often possible, to infer about the final response from the collected log events (see section 1.6, and Kroehne and Goldhammer (2018) for response-completeness of log data).

Motivation: Explicit automatic scoring of items can be necessary at runtime when scoring results that incorporate logical rules are required either for adaptive test assemblies (branched testing, multi-stage testing, or adaptive testing, see section 2.7.4), for the different feedback purposes (see section 2.9), or to monitor the test-taking processes. Scoring of CBA ItemBuilder Tasks is evaluated at task switches (i.e., when tasks are changed from one task to another). Task switches can either be triggered from within the Task (using Runtime Commands, see section 3.12, or from outside (by the deployment software, for instance, because of a global timeout, see section 7.2.8). Having the scoring definition implemented within the assessment components created with the CBA ItemBuilder can also simplify data post-processing workflows (see section 8.6) and sharing of items (for instance, as Open Educational Resources, see section 8.7.4). Hence, automatic scoring provides essential advantages over above mentioned alternatives: It standardizes the scoring procedures and gives immediate access to the scored results.

Implementing the scoring rules within the CBA ItemBuilder project files comes with a second advantage. The scoring definition becomes independent from the deployment software (see chapter 7) and the approach used for data post-processing (see section 8.4.2). Scoring embedded in CBA ItemBuilder projects can be tested already during item development and will be available after distributing items (as CBA ItemBuilder projects). An essential tool for checking the scoring in CBA ItemBuilder items is the so-called Scoring Debugger, as described already in section 1.5. The Scoring Debugger can be used to inspect the scoring live during Preview.

5.1 Terminology, Concepts and User Interface

The core component for the definition of the automatic scoring is the design of syntax conditions, which can be evaluated based on inputs (i.e., Component State of elements) and operators (i.e., incorporating the states in the internal finite-state machine(s) and the visited pages).

User Defined Id’s: The link between components and the syntax is provided by the User Defined Id's. Using the main menu Project > Edit all user defined IDs all named components of a CBA ItemBuilder Project File can be displayed. As shown in Figure 5.1, for a simple multiple-choice item, various components of type Checkbox can be defined, all of them with a unique UserDefinedId.

UserDefinedIds defined for example item shown in Figure 5.2.

FIGURE 5.1: UserDefinedIds defined for example item shown in Figure 5.2.

Figure 5.2 shows a simple item to illustrate the scoring of a multiple-choice item. The item was created by adding a HTMLTextField (see subsection 3.8.2 for the instruction and the four Checkboxes (see subsection 3.9.3). A simple scoring of this item might distinguish the conditions Correct (Jaguar and Panda) and Wrong (Jaguar and Panda are not selected or additional wrong options are selected). If a test-taker never selected any Checkbox at all, this might be called a Missing response (see section 5.3.11 for details).

FIGURE 5.2: Example for scoring a multiple choice item (html|ib).

The CBA ItemBuilder allows to define scoring-conditions using these UserDefinedIds. The conditions are labeled of Hits (i.e., Hit-conditions) and Misses (i.e., Mis-conditions) combining UserDefinedIds of components, and additional functional operators (see appendix B.2 for all operators), if necessary, combined with logical operators. Each condition represents a nominal conditions that is to be differentiated when computing the value of a scoring variable (called Class). The item contains one Task (defined in the Task-Editor, see subsection 3.6. The Task-Editor is also used to define the scoring (see Figure 5.3).

Task-Editor with one Task and three Hit-conditions for the item shown in Figure 5.2.

FIGURE 5.3: Task-Editor with one Task and three Hit-conditions for the item shown in Figure 5.2.

Class: Scoring is defined using nominal Hit-conditions. Each conditions corresponds to a potential value of a variable. To specify the relationship of values to variables, Hit-conditions (i.e., values) are assigned to Classes (i.e., Classes are equivalent to Variables in the final data set), as shown in Figure 5.4. The variable Score can have the three potential values Correct, Wrong and Omitted.

Class Definition Dialog for the item shown in Figure 5.2.

FIGURE 5.4: Class Definition Dialog for the item shown in Figure 5.2.

Name: Each hit- and miss-condition requires a unique name, that is defined in the Task-Editor (shown in Figure 5.3). This name represents the nominal value of the variable, if the corresponding condition is met (i.e., if the hit is active).

Condition Syntax: Each Hit-condition is defined by providing a scoring syntax. The example item shown in Figure 5.2 contains three different Hit, and the syntax for the conditions are shown in the editor for Conditions in Figure 5.5. The first condition for the hit Correct combines the User Defined Id's of the four Checkboxes with logical operators (using the CBA ItemBuilder specific bracketing of expressions, see section 4.1.3).

Condition Syntax for the Hit-condition Correct of the item shown in Figure 5.2.

FIGURE 5.5: Condition Syntax for the Hit-condition Correct of the item shown in Figure 5.2.

However, the syntax for scoring rules (see section 5.3) also allows using scoring operators, for instance, to incorporate information from the dynamic part of CBA ItemBuilder tasks. Operators are illustrated in Figure 5.5 in the syntax for the hit-condition for Wrong, which evaluates to true if the item’s current state is Answered. Note that the state Answered is implemented using a simple finite-state machine (see section 4.4 for details).

Scoring conditions can either be defined mutually exclusive (default, see section 5.3.3), or the order of conditions is incorporated (as used in the example in Figure 5.2). This suggested option is activated by selecting Use first active hit per class (applies to all tasks). (see right part of Figure 5.5).

Scoring Debug Window: The Scoring Debug Window (already introduced in section 1.5) can be used to explore the scoring of the example item shown in Figure 5.2, as shown in Figure 5.6. The Scoring Debug Window can be requested during item development in the Preview and is also available in the examples embedded in the online version of this book.

Screenshot of the Scoring Debug Window in a preview of the item shown in Figure 5.2.

FIGURE 5.6: Screenshot of the Scoring Debug Window in a preview of the item shown in Figure 5.2.

Multiple Classes: If components are to be used only in a particular way to form outcome variables, defining scoring constraints may be more onerous than strictly necessary (see section 5.2 for an alternative). The full potential of CBA ItemBuilder scoring unfolds in the use cases when different summaries of answers to variables are to be used. This is illustrated in Figure 5.7 for a simple Likert-style item. Assume that two variables should be created: One variable containing the response (Class: Response) and one indicating agreement or disagreement [Class: Style, as used, for instance, in models to investigate response style; cf. Böckenholt and Meiser (2017) and others].

FIGURE 5.7: Example for scoring a Likert-style item into two variables (html|ib).

By introducing the layer of hit definitions (described in detail in section 5.3), the CBA ItemBuilder allows the creation of flexible scoring for responses that can combine multiple components and can result in multiple variables (i.e., classes). Use cases not only include questionnaires (as the example in Figure 5.7) but can also be found for cognitive assessment, e.g., when both the raw response and the automatically scored response (correct vs. incorrect) are to be stored or when dichotomous and polytomous scoring is to be considered. Use cases for explicit scoring with multiple classes also arise if, for example, time measures are included in the scoring of responses.

Result-Text: Hit- and miss-conditions can be used to define evidence in terms of categorical values, which can then be assigned to classes to be used as outcome variables. However, also entered text and numbers are required as result variables. To copy text responses to result variables, the CBA ItemBuilder provides the result_text()-operator. The underlying idea is that each class (i.e., variable) can provide a Result-Text in addition to the name of the active hit. The condition defines which particular value is used as Result-Text. The (first) active hit-/miss-condition of a class defines which text is copied into the Result-Text.

The item shown in Figure 5.8 illustrates the use of the Result-Text. The first class (Var1) is used for question 1: Class Var1 has only one hit-condition with the syntax result_text(input1). This condition is always true, and whatever is entered in the SingleLineInputField with the User-Defined Id input1 is copied to the Result-Text for Var1. For question 2, the class Var2 is used with two hit conditions. When a text is entered into the InputField with the User-Defined Id input2 (i.e., the text is not empty checked with the condition matches(input2,"")), the value is copied to the Result-Text using the operator result_text(input2) in the condition Q2_Text. When nothing is entered, the Result-Text is filled with the string Missing (see hit-condition Q2_Missing). The class Var3 is used for question 3. The class contains either the selected option (A or B) in the Result-Text (see hit-conditions Q3_A and Q3_B). If neither A, B or Other is selected, the string Missing is copied to the * Result-Text* (see hit-condition Q3_Missing). Two hit-conditions are defined that deal with conditions that Other is selected. If no text is entered into the SingleLineInputField with the User-Defined Id input3, the text Other: Not Specified is copied to the Result-Text (see hit-condition Q3_OtherNotSpecified). If a text is entered, the Result-Text is filled with the string Other: followed by the provided text. This is achieved by using an argument list for the result_text()-operator (see 4.1.5). Note that Var3 will not contain the text entered into input3 if A or B is selected. This issue is addressed by defining Var4 that contains the text entered in input3, even if Other is not selected.

FIGURE 5.8: Item illustrating scoring with result_text()-operator (html|ib).

5.2 Scoring using FSM Variables

Using items provided by Toplak, West, and Stanovich (2014) the item in Figure 5.9 illustrates scoring using variables. In this example, the finite-state machine updates variable values, designed to allow immediate feedback (correct response, intuitive incorrect responses, and any other wrong response) and to compute the total score for all seven items of the Cognitive Reflection Test.

FIGURE 5.9: Example item illustrating scoring with variables (html|ib).

Starting with CBA ItemBuilder Version 10.4, a simple scoring definition is supported by providing default scoring for components (e.g., Checkboxes) and groups (e.g., RadioButtonGroup and FrameSelectGroups) using FSM Variables (instead of so-called Hit-/Miss-conditions).

5.3 Definition of Explicit Scoring Rules

5.3.1 UserDefinedId's as String Literals

An important part of possible scoring rules are input elements, i.e. components for the design of items, which have a value (i.e., are either selected or un-selected). To refer to the value of a component in a Hit- or Miss-condition, it is sufficient to include the UserDefinedId of the respective component into the condition-syntax. For instance, for a checkbox with the UserDefinedId: myCheckbox, the string literal myCheckbox is interpreted as TRUE if the checkbox is selected, when the syntax is evaluated. If the checkbox is not selected, the string literal myCheckbox is interpreted with the value FALSE.

The checked/unchecked state of CheckBox - components, the selected/unselected state of RadioButton - components, the toggle state of Buttons in toggling mode and the selected/unselected-state of ComboBoxItem in a ComboBox can be used to define Hit- or Mis- conditions by simply referring to the UserDefinedId of the component.

5.3.2 Syntax for Scoring Rules

The item scoring mechanism implemented in CBA ItemBuilder goes beyond simple mapping of Scoring Conditions (i.e., hit- and miss conditions) to component states. This is enabled by providing the possibility to formulate conditions as arbitrary combinations of statements using a so-called Domain Specific Language (DSL, i.e., by using a specific syntax).

To combine UserDefinedIds of the components to logical expressions, the following logical operators can be used:

  • A and B: true if A and B evaluate to true.
  • A or B: true if A or B evaluate to true.
  • not A: true if A is not true.

Flexible combinations of conditions are possible with the basic operators and, or and not. Use the Scoring Debug Window (Ctrl / Strg + S, see section 1.5) to explore the hit conditions in the item shown in Figure 5.10.

FIGURE 5.10: Hit definition with logical expressions (and, or, not; html|ib).

Notice the specific bracketing in the last hit condition shown in Figure 5.10: (((A and B) and C) and D). This condition illustrates that combining multiple Boolean expressions requires to include brackets so that the statement can be decomposed into pairs: A and B, (A and B) and C, and finally ((A and B) and C) and D.

It is important to note that the use of brackets is required to formulate statements with more than two conditions (see section 4.1.3).

For a number of scoring tasks, simply checking the Boolean value of components is not sufficient. Therefore, the CBA ItemBuilder provides functions in the scoring syntax (so-called operators), which can be used within the scoring syntax to take into account properties of the current task for the formulation of hit and miss conditions.

5.3.3 Sequential Evaluation of Scoring Rules

The CBA ItemBuilder provides two modes for evaluating scoring conditions within a class: the default mode (accumulated weights) and the sequential mode (first active hit/miss).

Default Mode (Accumulated Weights): By default, all Hit- and Miss-conditions are evaluated independently. If hits are assigned to classes, a central condition must be met: At any time, precisely one hit must be active for each class. This condition follows directly from using hits as (categorical) values for variables. Consequently, hit conditions within a class must always be formulated in such a way that they are mutually exclusive. To support checking this condition, the CBA ItemBuilder’s Preview of tasks provides the Scoring Debug Window, which contains a red exclamation mark once multiple hits are active for a class (see Figure 5.10). In the default mode, the result for each class is determined by the accumulated weight of all active hits.

Sequential Mode (First Active Hit/Miss): A recommended alternative is to activate the sequential evaluation of scoring conditions in the Task-Editor by selecting the checkbox Use first active hit/miss per class (applies to all tasks). If this option is activated, the evaluation is performed sequentially, starting with the first hit condition of a class. Only if the first hit condition is not true, the second hit condition is evaluated, and so on. The first condition that evaluates to true determines the result for that class—subsequent conditions are not evaluated. Accordingly, a last hit with the condition true can be added as a fallback to ensure that each class always has an active hit, which simplifies missing value coding (see the item shown in Figure 5.2 as an example and section 5.3.11 for details).62

The option Use first active hit/miss per class (applies to all tasks) is a project-wide setting that affects all tasks in a CBA ItemBuilder Project File. It is the recommended mode for all newly created items, as it simplifies the definition of scoring conditions and avoids the requirement for mutually exclusive conditions within each class.

Result Variables: The two evaluation modes produce different result variables in the scoring output:

  • In the default mode, the variable classHitWeighted.<ClassName> contains the accumulated weight of all active hits for that class. The value of a class is the name of the hit with the highest weight.

  • In the sequential mode, the variable classFirstActiveHit.<ClassName> contains the name of the first active hit in the class, and classFirstActiveMiss.<ClassName> contains the name of the first active miss. The Scoring Debug Window reflects this difference by showing the column header First Active instead of Weight and displaying the name of the winning hit directly.

In both modes, the Result-Text (see section 5.3.10) of the active hit is available as part of the scoring output.

5.3.4 Use of Text Responses in Scoring Rules

Text responses can be automatically scored inside of the CBA ItemBuilder using keywords or pattern. The provided matches()-operator takes two arguments: The UserDefinedId of the component used to collect the text response (see section 3.9.1) and a regular expression (see section 6.1 for details).

matches(UserDefinedId, RegularExpression)

The matches()-operator can be used with regular expressions (see section 6.1) and with concrete texts. Examples for using the matches()-operator are illustrated in Figure 5.11.

FIGURE 5.11: Different Hit- definitions using the matches()-operator (html|ib).

Note that the logical operators and, or, and not can be combined with several matches()-operators and other conditions. Hence, there is no need to formulate too complex regular expressions since multiple expressions can be combined using multiple matches()-operators.

5.3.5 Use of FSM-Variables in Scoring Rules

The value of FSM-Variables can be used within scoring rules (i.e., Hit- and Mis-conditions). This is achieved using the variable_in()-operator:

variable_in(FSMVariable,SetOfValues)

An examples for using the variable_in()-operator is provided in the Figure 5.12 for the scoring of a Drag-and-Drop response format, implemented using FSM Variables (see section 4.2.6 for the implementation of Drag-and-Drop).

FIGURE 5.12: Use of FSM-Variables in Scoring-Conditions with the variable_in()-operator (html|ib).

The visited_all_values_of_variable()-operator can be used to check whether a variable has taken one or more concrete values in the course of test-taking (see Figure 5.13 for an example):

visited_all_values_of_variable(FSMVariable,SetOfValues)

FIGURE 5.13: Using set of values and visited_all_values_of_variable()-operator in Scoring-Conditions (html|ib).

5.3.6 Use of Positions for Free Drag-and-Drop in Scoring Rules

As described in section 4.2.6, the CBA ItemBuilder supports free drag and drop. The panel_position_range()-operator can be used to score the position of drag-and-drop elements (see Figure 5.14 for an example):

panel_position_range(Container, [CheckNonMembers], XStart, XEnd, YStart, YEnd, 
                     Center, Component, Component, ...)

The operator evaluates to true if the (X,Y) positions of all given Components in the given Container are within the range given by XStart, XEnd, YStart and YEnd relative to the container’s (X,Y) position. If the flag CheckNonMembers is not give or set to true, the operator only evaluates to true if the (X,Y)-positions of all other components in the given Container are outside the given range. The upper left corner of the component is used as (X,Y)-position of a Component if the flag Center is not provided as true.

FIGURE 5.14: Example for scoring free drag and drop using the panel_position_range()-operator (html|ib).

Alternatively to the position of drag and drop element, the distance to score can also be evaluated. The panel_distance_range()-operator returns true if the mutual distance of all given Components in the given Container are within the given range between MinDistance and MaxDistance:

panel_distance_range(Container, [CheckNonMembers], MinDistance, 
                     MaxDistance, Center, Component, Component, ...)

5.3.7 Use of Events, States and Interaction in Scoring Rules

The CBA ItemBuilder provides various operators to incorporate events and the number of interactions into scoring conditions.

Number of Events: The number of events that have been raised during the execution of the current task can be used in scoring conditions. The CBA ItemBuilder considers an event to be raised even if it did not trigger a transition, and the count includes events raised by the raise()-operator:

raised_events()

If only the number of specific events should be counted, the raised_nb_events()-operator can be used:

raised_nb_events(SetOfEvents)

An even more advanced version of the raised_nb_events()-operator exist, that can be used to count how often one or multiple events were raised, while the item was in a particular state:

raised_nb_events_in_state(State, SetOfEvents)

Indicators for Events: In addition to the operators that count the events (of a particular type / within states), operators exist to check if an event was triggered. These operators evaluate to true (instead of returning the frequencies). The raised_all_events(EventA, EventB) return true if all events listed in the set of events (e.g., EventA and EventB) were raised:

raised_all_events(SetOfEvents)

Again, an more advanced version of the raised_nb_events_in_state()-operator exist, that can be used to check if one or multiple events were raised, while the item was in a particular state:

raised_all_events_in_state(State, SetOfEvents)

The following item shown in Figure 5.15 illustrates the use of the event-related operators.

FIGURE 5.15: Example for using events for scoring (html|ib).

Number of State Visits: The visited_nb_states()-operator return the number of the visits for a set of states during the execution of the current task:

visited_nb_states(State, State, ...)

Indicators State: The is_last_state()-operator returns true if the last state the finite-state machine is one of the given states in the SetOfStates:

is_last_state(SetOfStates)

While the is_last_state()-operator refers to the last state of the finite-state machine, the visited_all_states()-operator can be used the check if all states listed in the SetOfStates were visited during the execution of the current task:

visited_all_states(SetOfStates)

Number of Interactions: A simple generic operator is provided that counts the number of user-interactions within the current task:

user_interactions()

Note that this operator counts the total number of interactions within the running task. Counting specific interactions in FSM variables is possible using the finite-state machine (see section 4.4).

Elapsed Time: Another generic operator is provided that measures the elapsed time in the current task:

elapsedTime()

The scoring-operator elapsedTime() counts the total time in the current task. Measuring more specific time intervals is possible using finite-state machines (see section 4.4.6 and the example provided in Figure 4.60).

5.3.8 Use of Specific Operators in Scoring Rules

Tree Components: The scoring of response formats created using components of type Tree, TreeView and TreeChildArea (see section 3.9.9) is supported with the following operators:

  • The operator current_node() allows to check if in a particular Tree a RegularExpression matches to the node path ID of the current node:
current_node(Tree, RegularExpression)
  • The operator exists_nodes() returns number of nodes in the given Tree whose node path ID matches at least one of the given RegularExpressions (each node counts once only):
exists_nodes(Tree, RegularExpression, RegularExpression, ...)
  • The operator visited_nodes() returns number of visited nodes in the given Tree whose node path ID matches at least one of the given RegularExpressions (each node counts once only):
visited_nodes(Tree, RegularExpression, RegularExpression, ...)
  • The operator matches_nodes() returns the number of nodes in the given Tree whose node path ID matches the NodeIdPattern and whose column values match the specified ColumnPatterns. The first ColumnPattern corresponds to the node name, the second ColumnPattern to the first additional column, etc. (each node counts once only).
matches_nodes(Tree, NodeIdPattern, ColumnPattern, ColumnPattern, ...)

Pages: An operator current_page() is provided to check if a specified Page is currently displayed (or is displayed within the specified PageArea):

current_page(Page)
current_page(Page, PageArea)

For browser-components that support the bookmark function (see section 3.13.2) the following operator can be used to check if a page was bookmarked:

bookmarked(Page)

Spread Sheets: Operator to score value (or the computed formula) entered in a spreadsheet table with a given UserDefinedId (see section 3.9.8) as integer value:

integer_value(UserDefinedId, RoundingMode, Default)

The parameter RoundingMode can take the values up, down, half_up and half_down. If the text content is empty or does not represent a number, the Default value is returned.

To score the entered formula (instead of the value), the matches()-operator provides the additional argument Selector. If the value formula is requested, the operator evaluates the formula text of a spreadsheet table cell (instead of the formula value):

matches(Component, RegularExpression, Selector)

Highlighting: The following operators are provided to score the response format of multiple text highlighting (see section 3.8.3):

highlighted(RichText, RichText, ...)
complete(Selection, Selection, ...)
partial(Selection, Selection, ...)

5.3.9 Note on Scoring with PageAreas

PageAreas (see section 4.1.4) can be used to embed existing pages as content when designing pages. The CBA ItemBuilder allows that identical content can be re-used multiple times in different PageAreas on a single page, as illustrated in Figure 5.16. It is therefore generally necessary to add the UserDefinedId of the PageArea to all references to components displayed within PageAreas.

Components that are displayed in PageAreas need to be addressed using the following scheme: {UserDefinedId-of-PageArea}.{UserDefinedId-of_Component}.

FIGURE 5.16: Item illustrating scoring when PageAreas are used (html|ib).

5.3.10 Scoring Rules and Result Text

As shown in Figure 5.8, the CBA ItemBuilder integrates the handling of numerical and string responses into the Scoring Rules (i.e., Hit- and Miss-conditions) using the result_text()-operator. For each class, the active hit is determined first. If the option Use first active hit/miss per class (applies to all tasks) is activated (see section 5.3.3), this is the first condition within a class that applies. Otherwise item authors need to make sure that all conditions are mutually exclusive within each class. If the active hit contains a result_text()-operator, numerical or text input is provided as Result-Text.

5.3.11 Missing Value Coding for Tasks with Multiple Pages

The following examples shows, how to define scoring for single choice and multiple choice items including hits for not reached items and omitted responses. For this purpose, a variable is defined in the finite-state machine that counts how often a page was visited.

Items without response on a not visited page are coded as not reached (NR), missing responses on visited pages are coded as omitted response (OR). For more details, see the item shown in Figure 5.17 and use the Scoring Debug Window (as described in section 1.5.

FIGURE 5.17: Item illustrating Missing Value Coding for a multi-paged item (html|ib).

5.4 Automatically Generated Variables

The CBA ItemBuilder runtime will create some selected variables automatically:

  • reactionTime: Time (in milliseconds) between the start of the task execution and the first user interaction.
  • execTime: Time in (milliseconds) since the start of this task execution.
  • nbInteractions: Number of user interactions since start of the current task execution.

Deployment software for CBA ItemBuilder tasks (see chapter 7) can use identical tasks multiple times and can allow to re-visit tasks. For that purpose, the runtime also computes the cumulative variables:

  • reactionTimeTotal: Accumulated time (in milliseconds) between the start of the task execution and the first user interactions in previous executions of the task (excluding the last execution, that can be found in the variable reactionTime).
  • execTimeTotal: Accumulated time (in milliseconds) in previous executions of the task (excluding the last execution, that can be found in the variable execTime).
  • nbInteractionsTotal: Accumulated number of user interactions in previous executions of the task (excluding the last execution, that can be found in the variable nbInteractions).

5.4.1 Scoring Complete Tasks with Weights

This scoring is included in the CBA ItemBuilder to maintain compatibility with old items. It is considered outdated, as it only allows to derive one score per tasks.

If only dichotomous scoring is required for the complete Task, the CBA ItemBuilder implemented a simple approach.

  • MinHits: For each Task can be defined, how many Hit-conditions must be fulfilled, that the task is scored as True.
  • Weight: Each Hit-/ and Miss-conditions is assigned to a Weight.
  • Class: Each Hit-/ and Miss-conditions is assigned to a Class and to each Class either Hit-/ and Miss-conditions are assigned.

Each task provides the following results:

  • result: Overall result (\(1\) if the at least the number of hits is active that is defined as the property MinHits, \(0\) otherwise).
  • nb_Hits: Number of (active) hits.
  • Hit_weight: Total weight of hits.
  • nb_Misses: Number of (active) misses.
  • Miss_weight: Total weight of misses.
  • credit_Class: Name of the Class with the highest value. The value is computed as the sum of weight for all active Hits (in classes with Hits) or all active Misses (in classes with Misses).
  • credit_weight: Weight of the class with the highest class weight.

5.5 Checklist and Complete Workflow

As a summary the following list describes the typical workflow that is required for implementing automatic scoring in the CBA ItemBuilder:

  • Prepare the implementation of scoring by defining explicit User Defined IDs (see 3.7.4) for all components that should be used for scoring. It is not possible to define scoring using the automatically generated User Defined IDs that start with a $-sign.

  • Define a task as an entry point for the CBA ItemBuilder project. Since the scoring definition is done per task, a task must always be defined first (see section 3.6 for details).

  • When tasks are defined, define Classes. For each variable that should be included in the result data for a particular Task define one variable. For all newly created items activate the option Use first active hit per class (applies to all classes).

  • Define Hit-Conditions, that evaluate to true if the required conditions are fulfilled (see section 5.3.2. Order the Hit-conditions and add a default condition with the hit-syntax true as last condition. This will ensure that each class has one active hit (see section 5.3.3). For a usual workflow Miss-conditions are not needed, neither are Weights.

  • Extract string information using the result_text()-operator, if necessary. While hits serve as values of categorical variables, the Result-Text can be used to capture numerical or text responses.

  • Assign hits to classes. Classes fulfill the function of variables in the scoring of CBA ItemBuilder projects. Besides the unique name of the class (variable name), a description of the class can be entered in the Class Comment (variable description).

  • Decide how to handle missing response and implement, if necessary, additional Hit-Conditions for omitted responses and not reached questions (see section 5.3.11).

  • Test the scoring implementation using the Scoring Debug Window. If the option Use first active hit per class (applies to all classes) was not activated, make sure that exactly one hit (or miss) is active for each class at any point in time (see 8.4.2).

5.6 Scoring Quick Reference

This section provides a compact overview of the CBA ItemBuilder’s scoring system. For detailed descriptions, refer to the sections referenced above.

5.6.1 Scoring Structure

Scoring in the CBA ItemBuilder is defined per Task (see section 3.6) using Hit- and Miss-conditions that evaluate to true or false. Conditions are grouped into Classes, which serve as scoring variables in the result data.

Hit-Condition Syntax:
  component1 and component2                  (selected components)
  component1 and not component3              (with negation)
  matches(input1, "^correct answer$")        (text matching)
  result_text(input1)                        (capture text response)

Key Concepts:

  • Hit: A condition that evaluates to true when a desired response pattern is detected.
  • Miss: A condition that evaluates to true when an undesired response pattern is detected.
  • Class: A scoring variable that groups related Hit- or Miss-conditions. Each class produces one value in the result data.
  • Result-Text: Additional text output attached to a class, typically used to capture open text or numeric responses via the result_text() operator.
  • Result-Label: Additional label attached to a class via the result_label() operator.
  • Sequential Evaluation: When Use first active hit/miss per class is enabled (recommended), the first matching condition within a class determines the result.

5.6.2 Scoring Condition Operators

TABLE 5.1: Operators available in Hit- and Miss-conditions (see appendix B.2 for the complete listing).
Operator Syntax Description
Component state checkbox1 true if component is selected
Negation not checkbox1 true if component is not selected
Logical AND/OR A and B, A or B Combine conditions
matches() matches(input1, "^regex$") Text matches regular expression
matches() matches(cell1, "^regex$", "formula") Match formula text in spreadsheet cells
integer_value() integer_value(input1, "half_up", 0) Parse text as integer (with rounding mode and default)
highlighted() highlighted(text1, "#FFFF00") true if text is highlighted (optionally with specific color)
complete() complete(sel1, sel2) true if all listed selections are active
partial() partial(sel1, sel2) true if at least one selection is active
variable_in() variable_in(V_ans, 1, 2) true if variable has one of the listed values
current_page() current_page(page2) true if page is currently displayed
bookmarked() bookmarked(webPage1) true if page is bookmarked
panel_position_range() panel_position_range(p, 10, 100, 10, 100, true, comp1) true if component is within coordinate range
panel_distance_range() panel_distance_range(p, 5, 50, true, c1, c2) true if components are within distance range
elapsedTime() elapsedTime() > 5000 Elapsed time in milliseconds
user_interactions() user_interactions() > 0 Number of user interactions

5.6.3 Scoring Result Operators

TABLE 5.2: Operators for capturing results in Hit- and Miss-conditions.
Operator Syntax Description
result_text() result_text(input1) Copy component text to Result-Text
result_text() result_text("Answer: %1$s", input1) Template-based Result-Text
result_label() result_label("Category A") Set Result-Label for the condition
result_label() result_label("Score: %1$s", input1) Template-based Result-Label
trace_text() trace_text("scored") Write to trace log
trace_snapshot() trace_snapshot("scoring") Write to trace log with task snapshot

All result operators always return true and can be combined with other conditions using and.

5.6.5 Tree Scoring Operators

TABLE 5.4: Operators for scoring Tree components (see section 3.9.9).
Operator Syntax Description
current_node() current_node(tree1, "^nodePattern$") true if current node matches pattern
exists_nodes() exists_nodes(tree1, "pattern1", "pattern2") Number of nodes matching patterns
visited_nodes() visited_nodes(tree1, "pattern1") Number of visited nodes matching pattern
matches_nodes() matches_nodes(tree1, "nodeId", "col1", "col2") Nodes matching ID and column patterns

5.6.6 Scoring with getItemScore()

The getItemScore() operator can be used within the Finite-State Machine to query scoring results at runtime (e.g., for adaptive feedback):

getItemScore(TaskName, Calculation)
TABLE 5.5: Available Calculation values for getItemScore().
Calculation Returns Description
result 0 or 1 Overall result (1 if minimum hits reached and no active misses)
nb_hits Integer Number of active hits
hit_weight Integer Total weight of active hits
nb_misses Integer Number of active misses
miss_weight Integer Total weight of active misses
credit_class String Name of the class with highest weight
credit_weight Integer Weight of the highest-weighted class
nbInteractions Integer User interactions in current execution
execTime Integer (ms) Time since task start
reactionTime Integer (ms) Time to first interaction