5.3 Definition of Explicit Scoring Rules
5.3.1 UserDefinedId's as String Literals
An important part of possible scoring rules are input elements, i.e. components for the design of items, which have a value (i.e., are either selected or un-selected). To refer to the value of a component in a Hit- or Miss-condition, it is sufficient to include the UserDefinedId of the respective component into the condition-syntax. For instance, for a checkbox with the UserDefinedId: myCheckbox, the string literal myCheckbox is interpreted as TRUE if the checkbox is selected, when the syntax is evaluated. If the checkbox is not selected, the string literal myCheckbox is interpreted with the value FALSE.
CheckBox - components, the selected/unselected state of RadioButton - components, the toggle state of Buttons in toggling mode and the selected/unselected-state of ComboBoxItem in a ComboBox can be used to define Hit- or Mis- conditions by simply referring to the UserDefinedId of the component.
5.3.2 Syntax for Scoring Rules
The item scoring mechanism implemented in CBA ItemBuilder goes beyond simple mapping of Scoring Conditions (i.e., hit- and miss conditions) to component states. This is enabled by providing the possibility to formulate conditions as arbitrary combinations of statements using a so-called Domain Specific Language (DSL, i.e., by using a specific syntax).
To combine UserDefinedIds of the components to logical expressions, the following logical operators can be used:
A and B:trueifAandBevaluate totrue.A or B:trueifAorBevaluate totrue.not A:trueifAis nottrue.
Flexible combinations of conditions are possible with the basic operators and, or and not. Use the Scoring Debug Window (Ctrl / Strg + S, see section 1.5) to explore the hit conditions in the item shown in Figure 5.10.
Notice the specific bracketing in the last hit condition shown in Figure 5.10: (((A and B) and C) and D). This condition illustrates that combining multiple Boolean expressions requires to include brackets so that the statement can be decomposed into pairs: A and B, (A and B) and C, and finally ((A and B) and C) and D.
For a number of scoring tasks, simply checking the Boolean value of components is not sufficient. Therefore, the CBA ItemBuilder provides functions in the scoring syntax (so-called operators), which can be used within the scoring syntax to take into account properties of the current task for the formulation of hit and miss conditions.
5.3.3 Sequential Evaluation of Scoring Rules
The CBA ItemBuilder provides two modes for evaluating scoring conditions within a class: the default mode (accumulated weights) and the sequential mode (first active hit/miss).
Default Mode (Accumulated Weights): By default, all Hit- and Miss-conditions are evaluated independently. If hits are assigned to classes, a central condition must be met: At any time, precisely one hit must be active for each class. This condition follows directly from using hits as (categorical) values for variables. Consequently, hit conditions within a class must always be formulated in such a way that they are mutually exclusive. To support checking this condition, the CBA ItemBuilder’s Preview of tasks provides the Scoring Debug Window, which contains a red exclamation mark once multiple hits are active for a class (see Figure 5.10). In the default mode, the result for each class is determined by the accumulated weight of all active hits.
Sequential Mode (First Active Hit/Miss): A recommended alternative is to activate the sequential evaluation of scoring conditions in the Task-Editor by selecting the checkbox Use first active hit/miss per class (applies to all tasks). If this option is activated, the evaluation is performed sequentially, starting with the first hit condition of a class. Only if the first hit condition is not true, the second hit condition is evaluated, and so on. The first condition that evaluates to true determines the result for that class—subsequent conditions are not evaluated. Accordingly, a last hit with the condition true can be added as a fallback to ensure that each class always has an active hit, which simplifies missing value coding (see the item shown in Figure 5.2 as an example and section 5.3.11 for details).62
Use first active hit/miss per class (applies to all tasks) is a project-wide setting that affects all tasks in a CBA ItemBuilder Project File. It is the recommended mode for all newly created items, as it simplifies the definition of scoring conditions and avoids the requirement for mutually exclusive conditions within each class.
Result Variables: The two evaluation modes produce different result variables in the scoring output:
In the default mode, the variable
classHitWeighted.<ClassName>contains the accumulated weight of all active hits for that class. The value of a class is the name of the hit with the highest weight.In the sequential mode, the variable
classFirstActiveHit.<ClassName>contains the name of the first active hit in the class, andclassFirstActiveMiss.<ClassName>contains the name of the first active miss. The Scoring Debug Window reflects this difference by showing the column header First Active instead of Weight and displaying the name of the winning hit directly.
In both modes, the Result-Text (see section 5.3.10) of the active hit is available as part of the scoring output.
5.3.4 Use of Text Responses in Scoring Rules
Text responses can be automatically scored inside of the CBA ItemBuilder using keywords or pattern. The provided matches()-operator takes two arguments: The UserDefinedId of the component used to collect the text response (see section 3.9.1) and a regular expression (see section 6.1 for details).
The matches()-operator can be used with regular expressions (see section 6.1) and with concrete texts. Examples for using the matches()-operator are illustrated in Figure 5.11.
Note that the logical operators and, or, and not can be combined with several matches()-operators and other conditions. Hence, there is no need to formulate too complex regular expressions since multiple expressions can be combined using multiple matches()-operators.
5.3.5 Use of FSM-Variables in Scoring Rules
The value of FSM-Variables can be used within scoring rules (i.e., Hit- and Mis-conditions). This is achieved using the variable_in()-operator:
An examples for using the variable_in()-operator is provided in the Figure 5.12 for the scoring of a Drag-and-Drop response format, implemented using FSM Variables (see section 4.2.6 for the implementation of Drag-and-Drop).
The visited_all_values_of_variable()-operator can be used to check whether a variable has taken one or more concrete values in the course of test-taking (see Figure 5.13 for an example):
5.3.6 Use of Positions for Free Drag-and-Drop in Scoring Rules
As described in section 4.2.6, the CBA ItemBuilder supports free drag and drop. The panel_position_range()-operator can be used to score the position of drag-and-drop elements (see Figure 5.14 for an example):
panel_position_range(Container, [CheckNonMembers], XStart, XEnd, YStart, YEnd,
Center, Component, Component, ...)The operator evaluates to true if the (X,Y) positions of all given Components in the given Container are within the range given by XStart, XEnd, YStart and YEnd relative to the container’s (X,Y) position. If the flag CheckNonMembers is not give or set to true, the operator only evaluates to true if the (X,Y)-positions of all other components in the given Container are outside the given range. The upper left corner of the component is used as (X,Y)-position of a Component if the flag Center is not provided as true.
FIGURE 5.14: Example for scoring free drag and drop using the panel_position_range()-operator (html|ib).
Alternatively to the position of drag and drop element, the distance to score can also be evaluated. The panel_distance_range()-operator returns true if the mutual distance of all given Components in the given Container are within the given range between MinDistance and MaxDistance:
5.3.7 Use of Events, States and Interaction in Scoring Rules
The CBA ItemBuilder provides various operators to incorporate events and the number of interactions into scoring conditions.
Number of Events: The number of events that have been raised during the execution of the current task can be used in scoring conditions. The CBA ItemBuilder considers an event to be raised even if it did not trigger a transition, and the count includes events raised by the raise()-operator:
If only the number of specific events should be counted, the raised_nb_events()-operator can be used:
An even more advanced version of the raised_nb_events()-operator exist, that can be used to count how often one or multiple events were raised, while the item was in a particular state:
Indicators for Events: In addition to the operators that count the events (of a particular type / within states), operators exist to check if an event was triggered. These operators evaluate to true (instead of returning the frequencies). The raised_all_events(EventA, EventB) return true if all events listed in the set of events (e.g., EventA and EventB) were raised:
Again, an more advanced version of the raised_nb_events_in_state()-operator exist, that can be used to check if one or multiple events were raised, while the item was in a particular state:
The following item shown in Figure 5.15 illustrates the use of the event-related operators.
Number of State Visits: The visited_nb_states()-operator return the number of the visits for a set of states during the execution of the current task:
Indicators State: The is_last_state()-operator returns true if the last state the finite-state machine is one of the given states in the SetOfStates:
While the is_last_state()-operator refers to the last state of the finite-state machine, the visited_all_states()-operator can be used the check if all states listed in the SetOfStates were visited during the execution of the current task:
Number of Interactions: A simple generic operator is provided that counts the number of user-interactions within the current task:
user_interactions()Note that this operator counts the total number of interactions within the running task. Counting specific interactions in FSM variables is possible using the finite-state machine (see section 4.4).
Elapsed Time: Another generic operator is provided that measures the elapsed time in the current task:
elapsedTime()The scoring-operator elapsedTime() counts the total time in the current task. Measuring more specific time intervals is possible using finite-state machines (see section 4.4.6 and the example provided in Figure 4.60).
5.3.8 Use of Specific Operators in Scoring Rules
Tree Components: The scoring of response formats created using components of type Tree, TreeView and TreeChildArea (see section 3.9.9) is supported with the following operators:
- The operator
current_node()allows to check if in a particularTreeaRegularExpressionmatches to the node path ID of the current node:
- The operator
exists_nodes()returns number of nodes in the givenTreewhose node path ID matches at least one of the givenRegularExpressions(each node counts once only):
- The operator
visited_nodes()returns number of visited nodes in the givenTreewhose node path ID matches at least one of the givenRegularExpressions(each node counts once only):
- The operator
matches_nodes()returns the number of nodes in the givenTreewhose node path ID matches theNodeIdPatternand whose column values match the specifiedColumnPatterns. The firstColumnPatterncorresponds to the node name, the secondColumnPatternto the first additional column, etc. (each node counts once only).
Pages: An operator current_page() is provided to check if a specified Page is currently displayed (or is displayed within the specified PageArea):
current_page(Page)
current_page(Page, PageArea)For browser-components that support the bookmark function (see section 3.13.2) the following operator can be used to check if a page was bookmarked:
bookmarked(Page)Spread Sheets: Operator to score value (or the computed formula) entered in a spreadsheet table with a given UserDefinedId (see section 3.9.8) as integer value:
integer_value(UserDefinedId, RoundingMode, Default)The parameter RoundingMode can take the values up, down, half_up and half_down. If the text content is empty or does not represent a number, the Default value is returned.
To score the entered formula (instead of the value), the matches()-operator provides the additional argument Selector. If the value formula is requested, the operator evaluates the formula text of a spreadsheet table cell (instead of the formula value):
matches(Component, RegularExpression, Selector) Highlighting: The following operators are provided to score the response format of multiple text highlighting (see section 3.8.3):
highlighted(RichText, RichText, ...)complete(Selection, Selection, ...)partial(Selection, Selection, ...)5.3.9 Note on Scoring with PageAreas
PageAreas (see section 4.1.4) can be used to embed existing pages as content when designing pages. The CBA ItemBuilder allows that identical content can be re-used multiple times in different PageAreas on a single page, as illustrated in Figure 5.16. It is therefore generally necessary to add the UserDefinedId of the PageArea to all references to components displayed within PageAreas.
{UserDefinedId-of-PageArea}.{UserDefinedId-of_Component}.
5.3.10 Scoring Rules and Result Text
As shown in Figure 5.8, the CBA ItemBuilder integrates the handling of numerical and string responses into the Scoring Rules (i.e., Hit- and Miss-conditions) using the result_text()-operator. For each class, the active hit is determined first. If the option Use first active hit/miss per class (applies to all tasks) is activated (see section 5.3.3), this is the first condition within a class that applies. Otherwise item authors need to make sure that all conditions are mutually exclusive within each class. If the active hit contains a result_text()-operator, numerical or text input is provided as Result-Text.
5.3.11 Missing Value Coding for Tasks with Multiple Pages
The following examples shows, how to define scoring for single choice and multiple choice items including hits for not reached items and omitted responses. For this purpose, a variable is defined in the finite-state machine that counts how often a page was visited.
Items without response on a not visited page are coded as not reached (NR), missing responses on visited pages are coded as omitted response (OR). For more details, see the item shown in Figure 5.17 and use the Scoring Debug Window (as described in section 1.5.
Note that Hit- and Miss-conditions can be ordered in the Task Editor of the CBA ItemBuilder.↩︎