7.2 (Technical) Terminology and Concepts
Finding an appropriate test delivery software requires some technical considerations. Before describing the available tools, this section briefly describes some important technical terms and concepts.
7.2.1 Deployment Mode (Online, Offline, Cached and Mobile)
All assessments created with the CBA ItemBuilder require a web browser or browser component for rendering the generated HTML / JavaScript content. Web browsers are available on various devices, including desktop and mobile computers, cell phones, televisions, and game consoles. CBA ItemBuilder items can be used one way or the other on various platforms with built-in modern web browsers.
Online Deployment: The assessment is conducted in a supported web browser, constantly connected to the internet, and data are stored on a web server. Accordingly, some technology is required for hosting the assessment. The required hosting technology depends on the deployment software, including, for instance, R/Shiny (see sections 7.1 and 7.3) or PHP-based web hosting or cloud technologies (e.g., using TAO, see section 7.4), or any server-side technology (e.g., dotnet core used either directly or containerized for the software described in section 7.5).
Offline Deployment: The same software components as for the online deployment (i.e., the assessment is conducted in a web browser, together with a server component), but the server- and browser components run on the same device. Accordingly, items and data are stored only locally, and an internet connection is neither required to prepare nor run the assessment. Data are collected by copying the data from the local devices or by collecting the devices (e.g., thumb drives).
Cached Deployment: The assessment is conducted in a web browser connected to the internet while the assessment is prepared, but no (reliable) network connection is required during the assessment. Hence, data are stored locally, and after the assessment is finished, data are collected either online or by copying the data from local devices.
Mobile Deployment: Mobile deployment is a specific offline or cached assessment conducted on a mobile device. Since mobile devices are used, either a native app or a progressive web app is used (instead of a server), and test execution is possible even if there is no reliable internet connection (i.e., it is a cached deployment on mobile devices). When internet connectivity is available, data can be stored simultaneously online or transferred when connectivity is available.
Mobile deployment is specific in different ways. Firstly, no direct file access is typically possible on mobile devices (instead, data are transferred via the internet). Secondly, an app can stay accessible after installation, allowing the implementation of prompts or notifications that are either offline or triggered via so-called push notifications can try to attract the test-taker’s attention. Third, mobile devices are often equipped with additional sensors that can contribute to the collected data (and paradata) if incorporated by the app.
7.2.2 Browser Requirements and Availability of (Embedded) Content
Since assessments created with the CBA ItemBuilder are always displayed in a web browser or a browser component78, Cross-Browser Compatibility and Browser Requirements need to be considered.
Browser Requirements: Not all browsers support all features, adhere to the usual standards, show HTML, and similarly interpret JavaScript content. A typical approach is, therefore, to test the prepared assessment content in different browsers and then, if necessary, restrict the assessment to specific browser versions.
For rendering the CBA ItemBuilder generated content, the React framework is used internally in the current versions.79 React is supported by modern browsers, and so-called polyfills could be added for older browser versions.80 However, the deployment software used to combine CBA ItemBuilder tasks to tests might add additional requirements to be available to run assessments successfully. DIPF/TBA provides tools for online assessment (e.g., IRTlib, see section 7.5) that require a web browser that supports WebAssemblies. Using the TaskPlayer API, additional deployment software can be implemented with other server technology and for all browsers or browser-components supporting React.
Besides the deployment software, special attention regarding browser support is also required for content that is integrated into CBA ItemBuilder items via ExternalPageFrames (see section 3.14). These extensions, created by JavaScript / HTML programmers, are integrated into items using so-called iframes and must be checked to ensure that they function correctly in all browsers used for an assessment.
ExternalPageFrame).
Limitations, for example in playing videos, can also be caused by configurations of the web server (e.g. the configuration to support range HTTP requests).
Cross-Browser Compatibility: If the browsers used in online deliveries cannot be controlled and standardized, it is recommended to test the assessments in advance in different browsers. On Windows computers, the display and functioning of CBA ItemBuilder items in locally installed browsers can also be tested by using the regular Preview and copying the preview URL (see section 8.4.1). To check cross-browser comparability of CBA ItemBuilder deployments (see section 8.4.1), an online deployment must be prepared first (for instance, using the R/Shiny example described in section 7.1).
A very efficient approach to testing the compatibility of assessments in different browsers is to use third-party vendors that provide web-based access to different browser versions on different devices (cloud web and mobile testing platforms, such as BrowserStack, or similar services).
Availability of (Embedded) Content: Content embedded Using ExternalPageFrame must not only be working in the browser that are expected to be used for an assessment. The embedded material must also be available during the assessment. The use of Local resources is required for offline deployment. Moreover, network connection must be stable and provide enough bandwidth for resources included as External resources (see section 3.14 for the configuration of components of type ExternalPageFrame).
Special attention is required when multiple assessments share the same internet connection, For instance, it must be possible to load large audio and video files fast enough, even if, for example in a class context, all test-takers share a network access. Factors that influence the bandwidth are besides the storage of data (upload of log data, result data and data for recovery, see section 7.2.9) especially the download of media files (images, videos, audio files) that have to be transferred from the server to the client. The underlying Taskplayer API (see section 7.7) provides methods to implement the pre-loading of media files.
7.2.3 Fullscreen-Mode / Kiosk-Mode
Test security (see section 2.10) can translate into different requirements, particularly for high-stakes assessments. To standardize assessments, the assessment content should fill the entire screen if possible. Depending on the deployment method, this goal can only be approximated (although not guaranteed in certain browsers or on mobile devices) using Fullscreen-Mode.
Fullscreen-Mode: In online assessments, a full-screen display of assessment content can be possible in (regular) browsers on computers with desktop operating systems (Windows, Linux, macOS), and Fullscreen-Mode must typically be initiated by a user interaction. However, since test-takers can exit full screen mode at any time, the assessment software might hide at least the item content when full screen mode is exited.
Mandatory Fullscreen: For assessments on desktop computers, a requirement can be to ensure that the assessment is only displayed if the browser is in full-screen mode. Since even browser-based assessment software can, to some extent, diagnose that the current view is part of a full-screen presentation, assessment software can make the full-screen presentation mandatory. If a test-taker exits the full-screen presentation, different actions can be considered: Access-related paradata (i.e., a particular log event) can be created, a test administrator can be informed using, for instance, a dashboard (see section 7.2.4), or the assessment content could be faded-out, advising test-takers to return to the full-screen presentation before an assessment can be continued.
Kiosk-Mode: In offline deployments (or if a pre-defined browser is provided for online deployments), a so-called Kiosk-Mode prevents users (i.e., test-takers) from exiting the full-screen presentation of assessment content. Various browsers and add-on’s offer Kiosk-Mode for different operating systems. For instance, the offline player provided as part of the IRTlib-software (see section 7.5) provides a Kisok-Mode for Windows.
A free software solution for establishing test security on computers with Windows, macOS and iOS operating systems, which is used in a variety of application contexts, is the Safe Exam Browser (SEB). Since SEB is a (standard) HTML browser, which has only been supplemented with additional components and configurations for test security, this software can be combined in various ways with the CBA ItemBuilder deliveries.
7.2.5 Input and Pointing Device
Special considerations are required regarding the input devices used for point-and-click and text responses.
Touchscreen vs. Mouse-Click: Assessments on touchscreens (i.e., using touch-sensitive displays) require additional thoughts for at least two other reasons. First, it is inherent in touch operations that the place where the touch gesture is performed (using either a finger or a specific stylus) is not visible when the finger covers parts of the screen. Hence, if a small area needs to be clicked precisely, this property must be considered when designing the interactive item. Second, web browsers might treat touch events differently from click events. In particular, for content embedded as ExternalPageFrame, it is necessary to ensure that touch and click events are treated by the JavaScript / HTML5 content in all browsers as expected.
Touch devices might also, by default, use additional gestures, including pinch zoom gestures and, for Windows operating systems, the so-called charms bar (activated using a touch gesture over the edge of the screen).
Touchpad vs. Mouse-Click: If notebooks with touchpads are used, an external mouse might be considered for standardization purposes (i.e., to make sure that test-taker can answer the assessment regardless of their familiarity with touchpads), and touchpads might be deactivated to avoid distraction when typing and clicking is required.
(Onscreen) Keyboard and Text Entry: Text and numeric entries in assessments are usually recorded using a built-in or connected keyboard. Text input might automatically trigger an on-screen keyboard for devices with a touch screen. The operating system automatically displays the on-screen keyboard and then reduces the space available on the screen for the item content. In addition, the on-screen keyboard might allow the test-taker to exit Kiosk-Mode.
Careful usability testing (and if a Kiosk-Mode is planned, robustness checks regarding test security) are necessary, particularly on touch devices.
7.2.6 Authentification / Account Management
Regardless of the specific software used for test delivery, the following terms are relevant to web-based assessments.
Session: An essential concept for online assessments is the so-called session. Even if data collection takes place entirely anonymously (i.e., without access restrictions), all data belonging to one person should ideally result in one data record. Accordingly, the session is the unit that bundles the data that can be jointly assigned to one (potential) test-taker. With the help of client-side storage (i.e., session storage, cookies, or local storage), the ID of a session, which is usually created randomly, can be stored so that the data for an assessment, if interrupted and continued from the same test-taker, can be matched. Sessions IDs can be stored within a particular browser on a computer if the client-side storage is activated and the test-taker agrees to store client-side information.
Log-in/Password vs. Token: Authentication for online assessments can use Log-in and Password (i.e., two separate components) or a one-time Token (i.e., only one component). Often, there is no reason to arbitrarily separate log-in and password if the identifier is created as pseudonym only for a particular data collection. A distinction can be made with regard to the question of whether the assessment platform checks the validity of the identifier (access restriction) or if the identifier is only stored.
Protection of Multiple Tabs/Sessions: Finally, when a particular authentication is implemented, it must be considered that the log-in can potentially occur multiple times and in parallel. The delivery software might ensure that an assessment with one access (e.g., log-in/password or token) cannot be simultaneously answered and accessed more than once.
7.2.7 Task-Flow and Test Assembly
As described in section 2.7 items (i.e., in case of CBA ItemBuilder assessments Tasks) can be combined in different ways to assessments. Using the CBA ItemBuilder, the assembly of individual assessment components into tests is part of the test deployment software. An assessment component is any component that can be meaningfully used as component, including a log-in page, cover pages, instruction pages, the actual items or units, and exit or closing pages shown to test-takers.
Linear Sequence: The most common arrangement of assessment components is a Linear Sequence. CBA ItemBuilder Tasks can create a sequence that might contain items, prompts, feedback pages and dialog pages. As long as the same sequence is used for all test taker, all deployment software can be used that support linear sequences.
Booklets: If different sequences or combinations of CBA ItemBuilder Tasks are used (of which each target person works on exactly one), the procedure corresponds to so-called Booklets (or test rotations). Depending on the study design, the assignment of a test taker to a particular booklet or rotation is either done in advance (usually assigned to log-ins or tokens, see 7.2.6), or the assignment is done on-the-fly when a session is started.
Skip Rules: The deployment software for CBA ItemBuilder assessments is also responsible for omitting or skipping Tasks depending on previous responses. This functionality is only required for the test deployment software, if the skip rule can not be implemented within CBA ItemBuilder Tasks (using, for instance, multiple pages and either conditional links or the CBA ItemBuilder task’s finite-state machine).
Multi-Stage Tests and Adaptive Testing: The more complex and elaborate the calculations become, which are required for the selection of questions (or pages in a CBA ItemBuilder task), and the more items in total are available, the more complicated the implementation within a (single) CBA ItemBuilder task becomes. Accordingly, special delivery software is required to implement complex multi-stage tests and item- or unit-based, uni- or multidimensional adaptive tests using CBA ItemBuilder items (see section 7.5).
7.2.8 Time Limits across Tasks
Analogous to a sequencing of assessment content across individual parts (such as Tasks in the case of CBA ItemBuilder created assessments), the time restriction across tasks can be provided by a deployment software.
Time Measurement: The requirement of a time limit across tasks typically exists only for the actual items of a (cognitive) test, while before and possibly after the time-restricted part further content, for instance, with instruction pages and a farewell page are administered. Moreover, if a time limit across tasks is used, one or multiple additional pages are often shown, when the timeout occurred.
Remaining Time: When the minimal time to solve a particular item is known or can be approximated, the deployment software might apply the time limit already at a Task-switch (i.e., when in CBA ItemBuilder based assessments a new task is requested based on a NEXT_TASK-command, see section 3.12.2), if the remaining time (i.e., the time before the time limit will occur) is below the expected minimal time that would be required to work on the (requested) next task.
Result-Data (Post-)Processing and Timeouts: Time limits not only affect the presentation of items, meaning that within a particular section of the assessment no more items (i.e., Tasks within CBA ItemBuilder project files) are presented, when a timeout occurred. Time limits also affect the (missing) coding of the responses in the current task and in all subsequently tasks, not presented because of a timeout.
First, it is important to note, that the deployment software requests the ItemScore (see section 7.7) for the current item (i.e., Task within an CBA ItemBuilder project file) when a timeout occurs. Second, within this task, variables (i.e., classes) can be coded as Not Reached (NR), if the corresponding questions were not yet presented in a Task with multiple pages. Hence, the differentiation between Not Reached (NR) and Omitted Responses (OR) for Tasks with multiple parts must be included in the (regular) CBA ItemBuilder scoring (see section 5.3.11).
For assessment components not presented because of a timeout (or because an interviewer aborted the test, see section 7.2.4), no ItemScore will be available, because it is computed only when a Task is exited. Hence, the deployment software is required to determine the remaining tasks that would have been presented without the interruption and assign the appropriate missing code, either Not Reached (NR) in case of a timeout or Missing due to Abortion if an interviewer aborted the assessment.
7.2.9 Data Storage and Test-Resume
Regardless of the deployment software, there are some data types that arise in all CBA ItemBuilder-based data collections. These are briefly described in this section.
ItemScore (Result Data): When exiting CBA ItemBuilder items by a task-related command (see section 3.12.1) or by an external navigation request of the delivery environment (timeout or test abort), the defined scoring rules (see section 5.3) are evaluated. In the current version of the CBA ItemBuilder (i.e., for the REACT generator), the ItemScores are provided by the runtime as JSON data and collected by the delivery platform.
Traces (Log Data): In addition to the ItemScore data, trace data are provided automatically by the CBA ItemBuilder runtime. The data are provided as individual log events and can be collected by the deployment software or even be analyzed already instantly during the assessment.
Snapshot: In addition to the two data collected for further empirical analyses, snapshots of all internal states that represent the tasks are provided by the runtime so that even in case of interruptions, the possibly complex CBA ItemBuilder tasks can be continued at the last processing state.
Assessment content that is embedded into CBA ItemBuilder items using ExternalPageFrames can use the Snapshot of the CBA ItemBuilder Runtime to implement persistence of content across page changes (see section 4.6.6).
This is true even if specific browsers (such as the Safe Exam Browser) are used or if CBA ItemBuilder content is integrated with the TaskPlayer API in mobile applications (so-called hybrid apps) or in desktop applications (e.g., using Electron)↩︎
This is true for all CBA ItemBuilder versions starting with 9.0, see Appendix B.5 for details.↩︎
See, for instance, https://create-react-app.dev/docs/supported-browsers-features/.↩︎