7.5 Using CBA ItemBuilder Items with the IRTlib-Software
7.5.1 IRTlib-Editor vs. IRTlib-Player
The IRTlib Deployment Software consist of two different parts. An IRTlib-Editor to configure studies, and a IRTlib-Player to run studies. The IRTlib-Player requires a valid configuration of a deployment, that can be created, changed and modified with the IRTlib-Editor.
The IRTlib-Player supports different deployment modes for assessments, either offline (e.g., from USB stick) or online (e.g., stand-alone with a URL pointing to a dedicated server prepared for hosting the software).
The IRTlib-Editor used to configure and define studies can currently be use only offline.88 The identical assessment content (i.e., CBA ItemBuilder Project Files containing one or multiple tasks) can be used for online and offline delivery.
7.5.2 Runtime Requirements (Offline and Online)
An installation of the IRTlib-Editor is not necessary for offline use, because the software (i.e., the IRTlib-Player and -Editor) is prepared to run without installation (currently only under Windows).
Readiness Tool (offline): A simple command-line readiness-tool is provided to check if a (Windows) computer fulfills the requirements to run the IRTlib-Player for offline use. The readiness tool is configured with command-line parameters and, if so configured, also forwards the command-line parameters to the player. This allows the readiness-tool to always be started before the player, or to be called after the player only in the event of an error.
The IRTlib-Player for online deployments is provided as docker image. For preparation and configuration of online deployments, file management of CBA ItemBuilder project files and configuration files the offline IRTlib-Editor can be used.
Requirements (online): Unlike the CBA ItemBuilder runtime itself (see section 7.7) the IRTlib-Player for online deployments has additional browser requirements for online data collection. Online collections are possible in browsers that support the execution of WebAssembly (WASM).89
7.5.3 Study Configuration
Global configurations within the IRTlib-Editor concern the available CBA ItemBuilder Runtimes. Hence, before using the IRTlib-Editor for CBA ItemBuilder deployments it is necessary to add at least one Runtime. Moreover, for each study it needs to be configured, if either Login90 and Password or a access Token will be required.
Deliveries (i.e., studies) can be configured in the IRTlib-Editor. Studies need a unique Name and allow for an additional Label. Additional configurations for Studies include settings regarding the authentication of test-taker, the requested scaling of content, the configuration of a menu for test-administrators and the expected behavior, when a test-taker finishes a session. Finally, the IRTlib-Editor allows to define the Routing between Test Parts.
Each Study needs at least one Test Part, and currently two types of Test Parts are supported: CBA ItemBuilder and SurveyJS.
7.5.4 Configuration of CBA ItemBuilder Test Parts
To prepare a Test Part of type CBA ItemBuilder, the CBA ItemBuilder project files need to be added to the configuration using the IRTlib-Editor. CBA ItemBuilder Project Files (with Tasks and Scope) can be added at different sections of a Test Part configuration.
After adding a CBA ItemBuilder Project File the Tasks defined in that Project File are available in the IRTlib-Editor. The additional Scope allows to use an identical Task multiple times. The section info allows to add items (i.e, CBA ItemBuilder Project Files with Tasks and Scope) to the following slots:
- Prefix: Items administered in the beginning of a Test Part (e.g., a cover page, an introduction or tutorial etc.)
- Epilog: Items administered at the end of a Test Part (e.g., a thank-you page etc.)
- Timeout: If the Test Part is time restricted, one or multiple items can be added that will be administered, if a Timeout occurs in the Test Part
The main items that should be administered between Prefix and Epilog can be added to the section Items. If the option Routing is not activated, these items are administered in the sequence defined in the view Items. If Routing is activated, the View Routing allows to define the sequence of items (i.e., CBA ItemBuilder Project Files with Tasks and Scope) using the Visual Designer.
7.5.5 Configuration of Item Pools
For the definition of sequences, results of concrete items (i.e., Classes with scoring conditions and Result Texts, see chapter 5) can be accessed directly. For adaptive tests, in which the current item (or the list of current items) is selected using IRT-methods (see section 2.7), the mapping of the scoring defined within the CBA ItemBuilder Project Files and the variable \(u\) (e.g., \(0\) for no credit, \(1\) for full credit and \(0.5\) for partial credit), the IRTlib-Editor provides the additional concept of an Item Pool.
7.5.6 Configuration of Codebooks
Each item (i.e., a particular Task in a CBA ItemBuilder Project File administered in a particular Scope) provides a number of result variables (i.e. Classes, see section 1.5.1) as part of the ItemScore, when the presentation ended for a particular task either because of a navigation-related command (see section 3.12.2) or because of a time limit (see section 7.2.8). The Codebook allows to map the values provided by the CBA ItemBuilder scoring into variables with labels and value labels, as required for data sets.
Missing Value Coding: If a response was provided, either the name of the scoring condition (hit/miss) or the Result Text represents the value of the variable. Within the tasks, the defined scoring (see section 5.3.11) is expected to differentiate between Not Reached (NR) and Omitted Response (OR), if no response is provided. Hence, the values provided for classes (i.e., variables) of Tasks (i.e., for one or multiple items) are missing value coded based on the CBA ItemBuilder scoring. However, there are a few processing steps required by the IRTlib-Player to apply the missing value coding completely:
If a timeout or an abort by the test administrator / interviewer occurs, the classes (i.e., variables) within the current task (i.e., the task in which the test part was aborted or the task that was ended by a timeout) the Not Reached (NR) codes are changed to either Missing due to Abort or Missing due to Timeout.
In the next step, the remaining tasks are determined that would have been administered if the timeout or the abort would not have happened. The classes (i.e., variables) of these tasks are then also missing-coded as either Missing due to Abort or Missing due to Timeout.
Finally, all remaining tasks are determined (i.e., the tasks that are neither administered before the timeout or the abort and would have also not been administered if no timeout or abort had occurred). The classes (i.e., variables) for these tasks are missing-coded as Missing by Design.
Taking all three steps together the resulting data set of the obtained item score can be transformed into a regular rectangular data set, with one column for each class (i.e., variable), where the number of classes (i.e., variables and thereby columns) is known in advance as soon as all pairs of CBA ItemBuilder projects files and task names are defined and the CBA ItemBuilder project files are available.
For test designs with a unit structure (i.e., multiple items within a task), a differentiation between Not Reached (NR) and Omitted Responses (OR) needs to be included in the scoring definition of the CBA ItemBuilder Tasks. The IRTlib software then uses this differentiation to differentiate between normal responses, Missing by Design, Not Reached (NR) / Omitted Response (OR), Missing due to Abort, and Missing due to Timeout even in the case of complex task-flows or test assemblies that may depend on given responses, conditions and skip rules or even so-called pre-load variables (i.e., values assigned in advance to logins or tokens or values provided at the start of an assessment, see section 7.5.3).
7.5.7 Data Collected by the IRTlib-Software
Digitally-based assessments with CBA ItemBuilder Tasks create results data (based on the scoring definition, see chapter 5), raw log events (automatically generated log events), and additional user-defined log events (triggered either in the finite-state machine or in content embedded using ExternalPageFrames).
Raw Data Archives: The data stored by the IRTlib-Player always have the reference to the so-called Session. If a login or token is used as Person Identifier to authenticate test-takers, the raw data of a session are stored in a file {personidentifier.zip}. If there was no login in a particular session (yet), or if the study is configured to use a random identifier, the raw data archive will be named according to this (randon) ID. The raw data archive. is a zip archive that can contain multiple files. The files ItemScore.json and Trace.json contain the item scores and log data provided by the CBA ItemBuilder Tasks in JSON format. The file Log.json gathers additional log data provided by the IRTlib-Player and the file Snapshot.json contains all information required to restore the state of a Task (used, for instance, for crash recovery).
Monitoring Data: After the completion of an individual assessment, summarized status information can also be provided directly by the IRTlib-Player. Using this functionality, it is possible to react to key measures (e.g., particular time measures) already during data collection without the need to process raw data archive containing the log or result data. The status information can either be written to a file (monitoring file) or passed in an online assessment as part of a forwarding URL. Computation of monitoring information is defined in the visual editor.
The monitoring data are created as variables in the visual editor and are stored as list of variable-value pairs. In the JSON file the variable-value pairs are stored in the following form:
{
"ExampleDateTime": "2021-08-02T10:25:58.6209884+02:00",
"ExampleInteger": 42,
"ExampleString": "Any String",
"ExampleDecimal": 3.1415926
}Please note that the representation of the values in the JSON format differs slightly depending on the data type.
The online version of the IRTlib-Player and Editor as SaaS-solution is under preparation.↩︎
See the website caniuse for browsers supporting the use of WebAssemblies.↩︎
These example items for login (IRTLibLoginExample.zip) and exit (IRTLibEndItemExample.zip) can be used to design CBA ItemBuilder tasks for the IRTlib software.↩︎