CBA ItemBuilder Workshop: Session 01

Ulf Kroehne

Paris, 2022/11/07 - 2022/11/08

Session 01: Welcome

Overview Session 01

Motivation

• Advantages and Benefits of CBA
• Item Authoring and QTI
• Examples for Pages and Components
• Terminology and Glossary

Get Started

• Installation
• Preview

Questions and “The Big Picture”

• This two-day workshop is created by various small sessions.
• The first session starts with some background about computer-based assessment that should motivates us to take a closer loog to the CBA ItemBuilder.
• We will install the tool and become familiar with an important feature, the preview of the item content that is included in a CBA ItemBuilder project file.

Advantages and Benefits of CBA

• Standardization
  • Audio- / Video-files (for instance, in instructions)
  • Control about which content is displayed when

• Instant Scoring
  • Feedback (correct/missing responses), multiple attempts
  • Adaptive test assembly (multi-stage or adaptive testing)

• Validity and Innovative Items
  • Contextualization of assessments (authentic items)
  • Simulations / technology-enhanced items / game-based assessments

• Log-Data and Process-Indicators
  • Test-taking behavior can be logged at a micro-level
  • Construction and validation of derived process indicators

• First of all, we want to make sure that we are clear about why we are investing time and effort in computer-based assessment.
• We do this because we aim to make diagnostic meaningful use of one of the advantages of CBA.
• Various benefits of computer-based assessment can be grouped roughly into four areas.

Paper- vs. Computer-Based Assessment

Paper-based assessments (PBA)

• Tasks and material distributed in paper booklets
• Results gathered after scanning and coding the answers

• Anything in between? Black box!

Computer-based assessments (CBA)

• Can fill the missing blank, but we need to get rid of old habits

  • Provide information intentionally (e.g., duplicate digital content)
  • Learn tools to design complex computer-based assessments
  • Building a vocabulary to describe and understand test-taking behavior

• In early years of CBA, the comparison between PBA and CBA was helpful. Idea was/is to reflect on habits and traditions.
• But for PBA and early CBA implementations, often only the result is captured.Test-taking behavior and everything during the assessment was/is a black box.
• CBA can be more than a one-to-one replacement of identical content.
• Knowledge about possibilities required for item authors to

• Move from static items to dynamic environments designed to gather evidence for latent constructs

Psychometric Assessment

• Beyond multiple-choice (i.e., multiple ways to elicit diagnostic information)

• Beyond outcomes of single interaction only

  • Typical PBA’s were composed by a sequence of interactions
  • CBA allows more flexible designs than sequences of interactions
  • Boundaries between stimulus and response blur towards designed interactive environments

• Maintain Core Ideas of Psychometrics

  • Latent constructs / Item-Response Theory
  • Evidence identification (Assessment Frameworks)
  • Objectivity/Standardization, Reliability, Validity

• Aim for Openness

  • Standards, Open Science (e.g., Reproducibility), Open Source (e.g., Software, Scripts) and Open Educational Resources (e.g., Content)

Levels of Assessment Design (1)

A: Item-Level (i.e., Item and Task Design)

• Distribution of information (stimulus, tasks, pages)
• Visible components (pages) and defined behavior
• Specific stimuli (e.g., graphics, text, …) and response formats
• Navigation restriction and tools (e.g., pocket calculator)
• Forced choice and multiple attempts

B: Test Level (i.e., Booklet or Test Design)

• (Global) Time restrictions
• (Between-Unit) Navigation
• (Automated) Test assembly (ATA) / booklet designs

• B is not the main focus for the workshop.

Levels of Assessment Design (2)

A: Item Authoring Tool

• Items or units¹ created by content experts (= item authors)
• Content Experts create the interactive items, to make use of the available possibilities and features offered by CBA

Sure, content experts can develop innovative items without actually implementing them. However, knowing a particular tool’s possibilities and flexibilities can help design standardized assessment situations to gather authentic (and innovative) diagnostic evidence.

B: Delivery Platform

• Access (e.g., accounts), and time limits (at the test level or for sections)
• System performance (e.g., high availability), and security (e.g., kiosk mode)
• Sequences (e.g., linear) or adaptive testing (multi-stage or unit-/item-level)

• The CBA ItemBuilder is a tool for A.

1. Terminology: Depending on the concrete context (either single items, tasks, questions, units, testlets, etc.)

Approaches to Item Authoring

Online survey tools

• Limesurvey, soSci-survey, SurveyJS, Microsoft Forms, Google Forms, …

E-Learning tools

• Learning management systems (LMS, e.g., Moodle Quizzes), H5P, Adobe Captivate, …

TAO Assessment Platform

• QTI interactions to create items, Integration of IMS QTI/PCI-components

Curriculum- / domain specific tools

• GeoGebra, CindyJS, …

Psychological Research Tools

• PsychoPy, INQUISIT, …

• What computer-based assessment creation programs are you familiar with?

QTI Interactions

Various response formats, defined by QTI, e.g.,

• The CBA ItemBuilder can combine the single interactions and create multi-paged Task-environments (i.e., the complete interactive item in the Figure above is one CBA ItemBuilder Task).

• On this page you can see some interaction types defined by the QTI standard.
• We love standards! But: How to create technology-enhanced items within this strict set of interactions?

Pages & Components

Items are a combination of Pages, and Pages contain Components.

• Components exist for various purposes:

• Text
• Images
• Links
• Responses (1)
• (2)
• (3)
• … & More

• Components for Tables, Lists, and Menues
• Components for Audio and Video
• Components for Simulated Web Browsers
• Components for JavaScript/HTML5-Extensions, e.g., GeoGebra
• …

• CBA ItemBuilder is a tool to design items (using Pages and Components).

Workshop Goals

Learn how to use the CBA ItemBuilder to design Simple Items.

• Hands-on: Traditional response formats / single page items

Create items with Variables and Map-based Value Displays.

• Drag-and-drop response format
• Small scenarios with input and output

Create items with Conditional Links.

• Provide feedback, allow for multiple attempts

Get an overview about features for Dynamic Items.

• Design items using Pages (Page Areas ans Dialog Pages)
• Possibilities using Finite-State Machine

Terminology

• Assessments are composed by multiple Entities (items, units, task, …), which are presented either on a single pages or on multiple pages with separate areas (e.g., simulated hypertext environments).

• Entities are understood as independent parts of an assessments (i.e. they can be administered at different positions, for instance, in rotated booklet designs).

• In the following we will call the smallest entity that represent content that aloways belongs together CBA ItemBuilder Tasks.

The term Task is used as Technical Term. Hence a Task can be a single item, a unit, or even a complete test.

Project Files with Tasks

• Tasks are implemented in CBA ItemBuilder Project Files.
• Each Project File must contain at least one Task.

Like Microsoft Word or Open Office are editors for *.docx files, the CBA ItemBuilder is the editor for Project Files (and Project Files are stored as *.zip archives).

• When different Tasks use similar content (e.g., multiple units using identical multimedia resources or hypertext environments), one Project File can contain multiple Tasks.

• The distribution of assessment content to different Project Files is a decision based on a) the content and b) the intended use.

While many CBA ItemBuilder project files are typically used for a (e.g., large-scale) assessment, a signle project file is sufficient to create a PCI component.

Glossary

Definition / Description of Terms (see cba.itembuilder.de)

Installation

Steps

• Download and double-click CBAItemBuilder-9.8.0-windows-installer.exe

• Follow the Installation Wizard (Next … Finish)

The CBA ItemBuilder is available for Windows computers (mail to request the latest version).

Preview

Steps

• Download CBA ItemBuilder Project File (e.g., ExampleItemToPreview.zip)
• Start the CBA ItemBuilder application:

• Open the Project File (e.g., using main menu File > Open Project)

• Request a preview (e.g., using main menu Project > Preview Project)

• Keep settings in the dialog as is (Preview Task Task01) and confirm with OK.

CBA ItemBuilder project files are ZIP archives, often automatically extracted after download. Don’t unzip the content, the CBA ItemBuilder expects ZIP archives.)

Checkpoint

You should the the following item in your (default) browser:

Questions?

Section 1.4 describes the Preview - feature.

End of Session 01: The Big Picture

Step 1

• Use CBA ItemBuilder to create single Tasks or linear sequences of Tasks as CBA ItemBuilder Project Files.

Step 2

• A Converter¹ is provided to package CBA ItemBuilder Tasks as Portable Custom Interaction (PCI).
• Created PCI’s with CBA ItemBuilder items can be embedded into TAO to run assessments / collect data.

Step 3

• CBA ItemBuilder Tasks provide collect Log Data and, if defined, Scoring.
• The R Package LogFSM² can be used to analyze Log Data.

• Step 1 is the focus of this workshop (i.e., building Project Files with Tasks using the CBA ItemBuilder). Let’s start in session 02.

• Step 1 is focus of this workshop.
• Because of step 2 I will say some words about github at the end of the workshop.
• Regarding step 3 we will at least prepare the scoring of items.

1. URL: https://github.com/DIPFtba/fastib2pci

2. URL: https://github.com/kroehne/LogFSM