8.1 Planning of CBA Projects
Assessment projects can face time pressure if necessary steps for planning and preparing have either not been considered or if timetables and milestone plans underestimated requirements for necessary steps. However, time pressure at the end might be less likely if a systematic approach is followed.
8.1.1 Overall Planning and Preparation
Table 8.1 lists the initial steps that should be taken before implementing specific items or tasks for a concrete computer-based assessment.
| Overall Planning and Preparation |
|---|
| Domain Definition and Claims Statements |
| Content Specification |
| Feature Collection & Requirements |
| Software-Tool Selection |
Domain Definition and Claims Statements: As the first step of planning and preparing an assessment project, the construct domain to be tested needs to be defined and articulated. After naming and defining the domain to be measured, clear statements of the claims to be made about test-takers knowledge, skills and abilities (KSAs) are required.
Content Specification: The fundamental arguments that should be possible based on an assessment require validity-evidence based on the test content to support the specific interpretations and uses of test scores. This requires a precise content specification (i.e., test content and test format), including specifications on how the KSAs should be measured, which cognitive processes are required, and which item formats and response types are intended.
Feature Collection & Requirements: A systematic collection and documentation of all requirements that exist regarding item presentation and test delivery is suggested before selecting a particular assessment software. Planning for technology-based assessments (see, e.g., International Test Commission and Association of Test Publishers 2022) also includes considering how the use of technology can directly impact the assessment experience.
What if the required functionalities and features of the assessment software and the requirements for test delivery, analyses, and implementation of the computer-based assessment still need to be precisely described? In that case, creating storyboards and implementing minimal examples (as described in the 8.1.2 section) could help.
Software-Tool Selection: Selecting software components for the different parts is possible based on the collected requirements. Various tools might be appropriate for implementing the actual items, the assessment delivery, and the data processing during or after the assessments. If different tools are used, their interplay poses another requirement.
Some aspects for the selection of software components are:
- Features of the software: Items of which type can be implemented using the assessment software (e.g., specific response formats, support of QTI items, items composed by multiple pages etc.)? Is response times measurement required with an accuracy of milliseconds, or is a web-based implementation appropriate?
- License to use the software: How can the software be used for different parts of the assessment project, including the actual data collection, archiving of the instruments, and the dissemination of the developed CBA instruments?
- Interoperability and vendor lock-in: How can the assessment content be used if key stakeholders or project partners change?
- Support and Service Level Agreement: Is technical support for implementing the items and conducting the data collections available, or can a specific agreement be concluded?
- Runtime requirements for offline delivery: Is test delivery without internet access possible and which devices and operating systems are supported? How is it ensured that testing can be carried out without interruption in the face of incorrect entries and that it can be continued after system failures?
- Requirements for online assessments: Bandwidth for group testing, hosting requirements, number of concurrent supported test-takers, redundancy and backup strategy, supported browser versions?
- Software integration: If developers are involved, are they required to implement the complete assessment or only parts (e.g., specific content embedded using
ExternalPageFrames, see section 3.14, or the integration of CBA ItemBuilder items using the TaskPlayer API, see section 7.7)?
The personal abilities, resources, and skills of those involved in the project also play a not inconsiderable role in the success of CBA projects. Assessment projects often require competencies from different areas, which is an argument for interdisciplinary teams.
Programmers and software developers are only needed in this process if specific extensions in the form of ‘ExternalPageFrame’ content are required or existing HTML5/JavaScript content is to be integrated. Psychometricians (e.g., for scaling and IRT-based item selection), and system administrators (e.g., for hosting online assessments on in-house servers), may be needed to complete a team.
8.1.2 From Idea to Implementation
Once the process model for item creation and software selection has been decided upon, the individual items (in cycles, if necessary) are implemented using the steps shown in Table 8.2.
| Item Development |
|---|
| Item Story Boards and Item Writing |
| Minimal Examples and Item Computerization |
| Item Review and Modification |
| Scoring Definition and Scoring Testing |
| Item Tryouts (Cog-Labs / Field Testing) |
| Item Banking |
Story boards: A first step for the creation of more complex computer-based items are storyboards, which illustrate in the form of sketches in which sequence information is to be presented and answer options are to be given. For diagnostically meaningful assessment components, particularly the behavior by which test-takers should provide information about their competence or ability is of particular importance, i.e., which behavior or actions should be used for evidence identification. According to the possibilities of computerized assessments to create interactive, authentic, and complex tasks (cf. Parshall 2002), evidence identification does not have to include the work products exclusively. Still, it can also refer to features of the test-taking process (i.e., process indicators from the log data included in the measurement model).
Minimal Examples: Based on the initial ideas and storyboards, the functionalities and features required for designing the computer-based items can be derived. Typically, developing complex items to the end is not necessary to check whether a specific implementation is possible. Instead, so-called minimal examples, i.e., executable items that exclusively illustrate a particular functionality, can be used.
Feature-Complete Prototype: Based on the division of content into pages, reused page components, and dialogs, designing a prototype is suggested that is as complete as possible and that at least fully maps navigation within assessment components (i.e., within-unit navigation). This step is not necessary if items are created based on an already developed and tested template.
Production of Audio files, Images and Videos: For the production of authentic tasks, simulation-based assessment and the contextualization of assessment content, images, audio, and video files are often required (see section 6.2). These must be created as accurately as possible and across tasks, with consistent font sizes, colors, etc., and saved at the required size.
Item Computerization: Combining and merging the visual material of items with potential possibilities for the test-taker’s interactions (i.e., ways to respond to the assessment content) is a creative process that should result in opportunities to collect (valid) evidence about what test-takers know, can do, and understand. In other words, everything should be allowed that helps in making justifiable claims about KSAs.
In order to exploit the potential of computer-based testing for creating tasks that require specific construct-relevant test-taking behavior and that elicit informative evidence, two approaches are possible: A) Collaborative work in interdisciplinary teams (content experts and developers) and an iterative, agile approach for implementing, evaluating, and modifying computer-based tasks. B) Content experts learn and utilize tools to implement computer-based items directly, allowing them to play around with potential implementations and evaluate the impact on task characteristics and the interpretation of work products and test-taking processes.
ExternalPageFrame, then feedback and review rounds are recommended (approach A).
Item Review and Modification: Tasks and computer-based implementations of items are usually not created in one step. Instead, assessment components are typically reviewed after an initial draft and revised in review loops to improve and optimize them step by step.
Item Tryouts (Cog-Labs / Field Testing): After item development (and testing, see section 8.4), initial empirical testing in cognitive labs (so-called coglabs) or small-scale testing (e.g., with only one school class) is often helpful. Use cases for tryouts are to learn more about the comprehension and usability of new tasks or to (roughly) estimate the required processing time and task difficulty. The test deployment software described in chapter 7, for instance the R/Shiny package ShinyItemBuilder (see section 7.3), can be used to implement a tryout. If necessary, either a screen-recording software can be used to capture the detailed test-taking process, or the tryout can use the CBA ItemBuilder’s feature of collecting replay-complete log data (see section 2.8.3).92
Item Banking: The steps that individual items must go through describe a process from initial design, revisions, and tryouts to scaling and then the operational use of items in an automated or manual test assembly technique. At each stage, persons with different roles, such as item author, item reviewer, psychometrician, test developer, project manager, and others, can change an item’s status in a pre-defined workflow. Possible workflows include the dissemination or archiving of operational items and the long-term storage of items required for follow-up assessments, subsequent cycles, or linking or bridge studies. Moreover, the role of persons and the defined workflow also determine which actions, such as commenting on an item, moving it to the next stage, or bringing an item back to a previous stage (or even discarding an item draft), are possible. Hence, instead of managing items in files (or CBA ItemBuilder project files) and metadata about items in spreadsheets, Item Banking using, for instance, web-based software is possible.
References
Note that content embedded as
ExternalPageFrameis currently not included in the replay-completeness.↩︎