8.3 IT-Management of CBA Projects
Assessment projects are often not created by one person alone. If a complete computer-based instrument is developed, content experts, psychometricians, and computer experts are involved. In research contexts, practical implementation of assessment components is often supported or even delegated to student assistants. Besides, when complex assessment software, such as the CBA ItemBuilder, is utilized, user-support may be involved. Additional content experts may be asked, for instance, to review the developed items or tasks. And when data collection takes place in the context of more extensive empirical studies, professional survey institutes or data collection agencies are sometimes involved and contribute to the overall success based on their diverse experience. This large number of participants and the many individual decisions at the content level and the technical implementation of computer-based assessments quickly become quite complex. For this reason, it is recommended to use project management software whenever possible, as briefly described in the following section.
8.3.1 Use of Project Management Software
First of all, personalized user accounts are needed to use project management software for preparing computer-based assessments in teams or groups. The user accounts might be organized using groups with different roles and permissions. Personal accounts that are not shared between users are prerequisites for efficiently dividing tasks between people and assigning changes to specific users.
To distinguish and structure different phases of creating, testing, piloting, and delivering computer-based assessments, project management tools provide the concept of versions or milestones. The actual work steps are then divided into parts (issues or tasks) and managed in an issue tracker. Each topic or ticket can then be assigned to a user and processed by them individually or assigned to other users for work sharing. Observers can be registered, informed about the progress of the processing of a task. Typically, issues or tasks can be structured hierarchically and combined into superordinate work packages. Project management software can be used to process tasks using well-defined workflows (e.g., tickets of a particular category can be assigned to one of the states ‘new’, ‘in progress’, ‘review’, ‘feedback’, ‘solved’ and ‘closed’). Each ticket is typically dedicated to one separate topic, and the individual tickets can be assigned to milestones or versions. In that case, the progress within the issue tracker can be automatically used to create a roadmap that shows which work steps still need to be completed before the next milestone is reached. Finally, project management tools also provide assistance for knowledge management, for example, by providing Wiki pages or additional storage for documents or files.
Open-source tools that can be hosted on your own servers are, for instance, Redmine (Lesyuk 2013; Pavic 2016), OpenProject or Gitlab (Hethey 2013; Baarsen 2014). If the assessment content does not require special protection, public cloud solutions such as Github can also be used (Tsitoara 2020).
8.3.2 Use of Version Control Software
First, the answer to the obvious question: Why is version control helpful for developing computer-based assessments? The explanation refers to the nature of computer-based assessments. Computer-based assessments are created using different software tools. A deployment software (see chapter 7) is used, for instance, together with a Web Browser or a browser component, to show the assessment to test-takers or participants. Moreover, the test content is either created with an authoring software for the test content (such as the CBA ItemBuilder) or is specifically programmed (i.e., implemented with a particular programming language). Altogether, all components together create the computer-based assessment, typically stored in multiple files. Accordingly, a tool that keeps track of all files and identifies differences between files and folders is useful for preparing computer-based assessments. Techniques that originated in software development has proven useful for managing and creating assessments. Version control software allows one or more users to manage and document the status of a set of files (referred to as a repository) and make changes traceable at the level of individual files. Moreover, a verbal description for changes in files is documented using so-called commit-messages.
Since item authors and users of the CBA ItemBuilder may not be familiar with the idea and concrete implementation of version management software, we describe two popular systems in more detail below. However, version control software is also a key component in the context of Open Science and Reproducible Research (Christensen, Freese, and Miguel 2019). The development environment for R (RStudio) for example comes with Git integration (Gandrud 2020).
8.3.2.1 SVN / Subversion
Often, project management software can be configured already to support the creation of repositories for version management. A project can then be assigned one or more repositories, each of which supports a specific version management technique. The use of two such methods will be described in some detail: Subversion (SVN) and Git.
Overview: Subversion (SVN) is less complex than GIT, which is why we describe this method first. Version management generally means that all files and documents belonging to a project are stored in a common repository. In the context of SVN, this repository is always on a server, i.e., usually accessible via a network. Only selected users have access to the repository. If SVN is used in combination with a project management tool, all registered users with the appropriate permission can access the repository. Every file is stored in the central repository in all created versions. Files can be added to the SVN repository using the commit operation. Check-out or update transfers or updates the files from the central repository to a local working copy (see Figure 8.3).
FIGURE 8.3: SVN: Repository and Working Copies (Mason 2006).
All users of an SVN repository work with local copies and can adjust, modify, delete, and add new files (or folders) in the working copy of the repository. After completing a particular step (e.g., after changing the assessment according to a specific ticket from the issue tracker), the changes are submitted collectively to the repository (i.e., checked-in with the help of a commit command). For that purpose, the individual changes are described in a commit message so that it is traceable which changes were checked-in. Suppose the changes are related to a specific ticket. In that case, the ticket number can be specified in the Commit message, for example, to document the changes in a traceable manner.
Repository URL: A repository URL identifies SVN repositories. If a public domain is used, they are thus globally unique. For example, the repository URL could look like this: https://{example-domain.org}/svn/{project-title}
If the version management is integrated into a project management software, then the repository URL can usually also be retrieved there.
Revision: To manage the different versions, SVN uses the concept of revisions. A repository is always in a concrete revision (starting with zero). Each commit, i.e. all changes to one or more files that are submitted at the same time, increases the revision number by one. The SVN repository stores the complete history of changes, i.e. for each file its content can be exactly determined (and if necessary also restored) to a specific revision. If you know the repository URL of an SVN and a specific revision number, then the content is also uniquely referenced.
Requirements: Many project management tools allow you to view a repository online in the browser and, for example, browse through revisions and, if necessary, follow the links to the issue tracker. However, client software is required to work with an SVN repository locally.
Different software tools for the different operating systems can be used to work with SVN repositories. For Windows, for example, TortoiseSVN is quite widespread, a free edition of an SVN software for MacOS offers, for example, SmartSVN.
Check-out: After installing an SVN client (e.g., TortoiseSVN), a working copy can be created locally for an existing repository. Even for a repository that is empty until then (i.e., revision 0), working with an SVN starts with a first check-out. The first check-out will create the working copy locally and connects the local folder to the SVN repository. If the repository is already filled with files and folders (i.e., in a revision greater than zero), then all files are downloaded and cached locally during the check-out. As soon as the check-out is completed (at a particular revision), it will be possible to work, modify, and, if necessary, even execute files in the working copy.
Checking out an SVN repository works the same way if there are already files in the repository.
Commit: The local working copy of an SVN repository can be worked within the same way as any other directory. After an intermediate (completed) state is reached, the files can be committed to the repository via the Commit command. For this purpose, the SVN client displays the files that have been changed. The selection to be transferred can be made and described with a commit-message. Afterward, the changed files are transferred over the network, and the revision number of the SVN repository increases by one. Newly inserted or files that were not yet under version control must be added to the repository with Add.
Save changes (for instance, in the CBA ItemBuilder) before committing files.
Update: As soon as more than two working copies are used (e.g., because several people are involved in the preparation of a computer-based assessment), the current status of a working copy may be out of date. For SVN repositories, this means, in the simplest case, that the current revision of the working copy is smaller than the most recent revision on the server (in the repository). If no files have been changed in the outdated working copy, a simple update can be used to update the working copy.
Check for Modifications: The question of whether files or directories in a local working copy have changed, been added, or deleted can be easily checked with the help of SVN. For this purpose, the function Check for Modifications is available, with the help of which a comparison of the working copy can be displayed with its current revision. This function can also be used to check whether an existing working copy still contains modified files that are not yet under version control.
Conflicts: As long as parallel changes in the SVN repository always affect different files, Commit and Update allow all users of the SVN repository to edit files in parallel and to share them using the repository. However, if two users make parallel changes to one or multiple identical files, so-called conflicts occurs. Conflicts can be related to files (file conflict) or to the directory tree (tree conflicts). Conflicts occur when executing the update command. If user A tries to commit a file that was changed and committed already by a different user B, SVN requests to update the working copy for user A if he or she tries to commit changes. Since the SVN repository is agnostic against its content, conflicts need to be resolved by users. With existing conflicts, no commits are possible. A graphical user interface (or the explorer integration of TortoiseSVN using the context menu) is of great value for resolving conflicts.
Advanced Features: A complete introduction to all features, options, and possibilities of version management with SVN would go beyond this book’s scope. Therefore, only the keywords for selected advanced features will be mentioned and briefly explained in the following:
Merging: SVN attempts to combine changes in the repository with local changes that have not yet been committed when updating. This process is called merging. If this does not succeed, a conflict occurs.
Ignored Files or Folders: Files or directories that should not be part of the repository but are located within the working copy’s directory can be excluded from the SVN. For instance, this function is useful if an assessment software that is part of the SVN, result data are written into a subdirectory, and test possible test data should be excluded from the repository.
Diff: For files in text format (i.e., explicit text documents, but also files with program syntax and files in CSV, DAT, INI, XML, JSON, YAML, and similar formats), the difference from a previous version can be easily displayed directly in SVN. The display of differences (diff) is beneficial, especially when the commit-message is not meaningful. SVN client tools (such as TortoiseSVN) create diffs for other file formats. Unfortunately, the simple visualization of differences of non-text based file formats (i.e., images, but also ZIP archives and CBA ItemBuilder project files) is often not possible.
Revert: In order to restore a previous state of the SVN repository, the revert function can be used to restore an earlier revision of the files and folders of the repository.
Version management using SVN can also use tags and branches, and has a concept for locking of files (see for more details, for instance, Mason 2006).
Summary: Version control allows to prepare and develop computer-based assessment using multiple files in repositories. The critical benefit of using version control compared to other file sharing approaches (e.g., cloud storage) is that modification of files are documented (using commit messages) and that conflicts (i.e., modification of identical part of the repository by multiple users) are detected (and infrastructure to handle conflicts is provided). Moreover, the revision number (of SVN repositories) can be used to exactly define the version of all93 files used for deployment of a computer-based assessment (i.e., for a particular data collection).
8.3.2.2 GIT
As a more modern alternative to SVN, the basics of version management with Git is now briefly described.
Overview: Git, unlike SVN, is a distributed version management system. This allow to use Git to manage different versions before pushing changes over a network to a remote repository. This two-step process of commits adds some complexities compared to SVN. However, it allows using Git (i.e., to perform almost all operations) locally.
FIGURE 8.4: GIT: Remote Repository and Local Repositories.
Repository URL: For GIT, the remote repository is addressed via a URL. How the URL looks exactly depends on the way of communication with the server. Possible protocols are https and ssh.
Commit Hash: Instead of an incremental revision number (that is used by SVN), each commit in git is identified by a SHA1 hash. To identify a commit, the hash shortened from 40 characters to 6-8 characters is often displayed in the git history.
Instead of taking the largest revision number, git uses a HEAD as a named pointer to a specific commit, representing the current commit (of a given branch).
Requirements: Various cloud services (e.g., github) and project management tools offer the possibility of creating git repositories and viewing them in the browser. Files can often also be edited or uploaded in the browser and changed directly in the repository via commit.
Git clients for all platforms can be downloaded from https://git-scm.com). Git clients are directly integrated in a number of tools (e.g., RStudio) and there are graphical tools for git (e.g. GitHub Desktop, SourceTree, and many more) as well as a Windows Explorer integration (TortoiseGit).
A full introduction to git is not necessary for organizing assessment projects and is beyond the scope of this chapter (see, e.g., Tsitoara 2020). In the following, only the basic steps necessary to use git without branches to manage files will be described.
Clone: Before working with files in a local working copy of a git repository, a copy of the repository is required. This can be created for empty and already used repositories via the Clone command. Compared to SVN (where checkout was used for this step), git uses clone to download not only the current commit (HEAD) but also all previous versions of files and all changes in the local repository.
Staged Files: Files within the working copy are, analogous to SVN, not automatically part of the repository. For that they have to be added with Add. Git then differentiates between the states for files shown in Figure 8.5.
FIGURE 8.5: Lifecycle of file status in git.
New files are initially ignored by git, i.e. their contents are untracked. When a file is added, git marks it as staged. A snapshot of the staged files can be created in a git repository via commit. After that the state of the committed files is unmodified until they are edited or changed. Then they are marked as modified. Before edited files with the status modified can become unmodified files again via Commit, they undergo the status of staged files again. Unmodified files can be removed from the repository (i.e., marked as untracked).
Commit: Tracked files that have been modified or added (i.e. files in the staged status, see figure 8.5) can be commented. Graphical tools for git often show the files in staged state or allow to stage all files that have been modified by a simple selection.
Analogous to SVN, a commit message is required for each commit, which is then used in the git history to describe the changes. The snapshot of the files from the working copy created with the help of a commit is marked with a hash at git and stored in the local repository.
Push: After committing files the additional push command is required to transfer the commit, which is made on a local branch of the git repository to a remote repository (see Figure 8.4).
Fetch: When multiple users push to a git repository, commits made by another user can be retrieved with Fetch. This makes the repository aware of changes, but Fetch does not yet integrate them into the local working copy.
Pull: Only with the command Pull will changes pushed to the repository by other users be downloaded and copied to the local working copy. If conflicts occur, these must be resolved and solved via a merge commit. Git provides the options use ours and use theirs for this.
Additional Features: The git tool, popular in software development, goes far beyond the features and functionality needed to manage (binary) assessment components in the form of CBA ItemBuilder Project Files.
Tagging: Since each commit is only marked with a hash, this is not well suited for naming a specific version (e.g. the tested final version of an assessment). For this purpose, the option of
taggingcan be used in git, where a concrete state of a repository is named with a (readable) label (e.g.v1.0).Branches: Git has a sophisticated concept for Branches, i.e. for the division into several areas, in which files with the same name can have a different status. For example, when changes are made by different users at the same time, git automatically creates Branches. Branches can also be used to test and develop changes in a protected section, while the main section remains usable. Often, Branches are used systematically with git, for instance, whenever a new feature is to be developed and tested. A popular strategy for this approach is git flow.
Revert and Reset: Git provides several ways to access previous commits in a repository. Especially when several users work in a common git repository, it is necessary to choose carefully here. In order to track the history of CBA ItemBuilder Project Files and to revert to a previous version if necessary, it is often sufficient to display the git history in the browser if the changes were committed to the central repository via `push’. Project Files in earlier versions can then be downloaded and reused if changes are to be discarded.
Similar to SVN, git can also ignore individual files or files of a certain type or in a certain directory via the
.gitignorefile.
Summary: Git is a complex and powerful version control system whose basic features can also be used to manage assessment projects. It is superior to SVN for this task if the versions are not stored in a central repository and are to be managed locally even without a network connection.
8.3.3 Working with Project Files as ZIP Archives
Extract CBA ItemBuilder Version: Each CBA ItemBuilder file contains a file {Project-Name}.json. In this JSON file, which can be read with a text editor, the supported version of the CBA ItemBuilder (runtimeCompatibilityVersion) is directly in the first line.
Extract Scoring Information: Suppose many CBA ItemBuilder files are to be tested automatically and integrated into delivery software. In that case, it is a good idea to automatically check the transfer of result data (i.e., the scoring). For this purpose, the scoring can be read out automatically from the JSON file included in the CBA ItemBuilder project files.94
Reading Metadata: CBA ItemBuilder Project Files contain metadata for describing the content (see section 6.3.4). Metadata can be found in the file metadata.xml. This XML-file following the Dublin Core specification can be extracted from the ZIP archives.
| Folder/File | Description |
|---|---|
stimulus.json |
Metadata and information about the tasks |
config.json |
Runtime definition of the tasks |
resources/ |
Resource files required for the task |
external-resources/ |
External resources required for the task |
Runtime Code: The CBA ItemBuilder Project Files fulfill two functions. They allow modifying, editing, and previewing of items using the CBA ItemBuilder desktop program. In addition, they can be used with existing deployment software (see, for example, section 7.5) or by programmers using the TaskPlayer API (see section 7.7) to execute assessments. Files and directories required at runtime are listed in Table 8.4. Files and directories required for editing with the CBA ItemBuilder are summarized in Table 8.5.
| Folder/File | Description |
|---|---|
metadata.xml |
Metadata defined in the Project File (see section 6.3.4) |
internal.json |
Internal project information used by the CBA ItemBuilder |
global_1_1.xlf |
XML file containing texts for translation in XLIF-format |
global.cbavaluemap |
XML file containing the definition of Value Maps |
global.cbaitemscore |
Scoring definition (with reference to dsl-file in folder scoringResources/ |
scoringResources/ |
Folder with dsl-files containing Scoring Conditions |
global.cbavariables |
XML file containing the definition of Variables |
global.cbalayoutsettings |
XML file containing the layout definition for Tasks |
conditionFiles/ |
Folder with dsl-files containing Conditional Link Conditions |
global.emfstatemachine |
XML file containing the State definitions |
statemachineFiles/ |
Folder with dsl-files containing Finite-State Machine syntax |
{page}.cbaml/.cbaml_diagram |
Page definition edited with the CBA ItemBuilder for each {page} |
project.properties |
Global properties of the Project File |
.project |
(Can be ignored.) |
resources and external-resources and make backup copies in any case, or use version management (see section 8.3.2).
Replace Resource Files: Images, videos and audio files added to CBA ItemBuilder Project Files via the Resource Browser (see section 3.10.1) are included in the ZIP archives in the sub-directory resources. If the file names (incl. upper and lower case) and the file formats (incl. the file extension) remain identical, resource files can be exchanged, modified, and updated even without the CBA ItemBuilder in the ZIP archives. The resolution (pixel width times height) for image and video files should remain the same to guarantee that the resources are appropriately rendered during runtime.95
Add ExternalPageFrame Resources: If in CBA ItemBuilder Project Files content is inserted as ExternalPageFrame, then the files are included unchanged in the directory external-resources of the ZIP archive. Content can be added to ZIP archives (i.e., CBA ItemBuilder Project Files) using the Embedded HTML Explorer (see section 3.14.2). As long as the Page Address (see Figure 3.152), i.e., the file which is directly included by a component of type ExternalPageFrame remains identical, the external resources in the directory external-resources can also be updated, added or inserted directly in the ZIP archive.
Edit Value Maps: Defining complex Value Maps using the editor provided by the CBA ItemBuilder (see section 4.2.4) can be cumbersome. The definition of Value Maps is stored in the file global.cbavaluemap inside of the CBA ItemBuilder Project Files. The following XML shows the content of the file global.cbavaluemap used for the example item shown in Figure 4.14 (see section 4.2.4).
<?xml version="1.0" encoding="UTF-8"?>
<valuemap:ValueMapper xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:valuemap="http://valuemap.softcon.de" version="VERSION_01_01">
<valueMaps xsi:type="valuemap:DiscreteValueMap" name="M_Example">
<entries guard="1" text="Red" image="1.fw.png"
audio="red.mp3" video="red.mp4"/>
<entries guard="2" text="Red-Yellow" image="2.fw.png"
audio="red-yellow.mp3" video="original.ogv"/>
<entries guard="3" text="Green" image="3.fw.png"
audio="green.mp3" video="green.mp4"/>
<entries guard="4" text="Yellow" image="4.fw.png"
audio="yellow.mp3" video="yellow.mp4"/>
</valueMaps>
</valuemap:ValueMapper>As long as the structure of the XML file remains valid and all resources exist (i.e., images, audio and video files mentioned in the XML attributes are included in the Project File), the file global.cbavaluemap can be extracted from the ZIP archive, edited in a text editor (or XML editor) and copied in the ZIP archive again. To make sure the CBA ItemBuilder Project File remains valid, open the file in the CBA ItemBuilder and preview all Tasks after changing the file global.cbavaluemap.
8.3.4 Use of Continuous Integration/Continuous Delivery
Continuous Integration (CI) and Continuous Delivery (CD) refers to techniques developed in software engineering, for automatically updating software environments enforcing automation in building, testing and the deployment of applications. CI/CD is organized in so-called Pipelines, which are code scripts executed based on triggers, such as Push commands in git repositories (see section 8.3.2.2).
Content created as Project Files with the CBA ItemBuilder can be used together with the TaskPlayer API (see section 7.7) in CI/CD-pipelines. The converter described for the integration of CBA ItemBuilder items as PCI components into TAO (see section 7.4) is based on a Github workflow. The pipeline that is shared as a Github template project (see the project fastib2pci) is configured to automatically build the PCI components using CBA ItemBuilder Project Files committed to the corresponding git repository.