Posts – Alex Wainwright

Extending CAT to Polytomous Items with adaptivetesting

Psychometrics

CAT

IRT

Simulating a polytomous CAT using the adaptivetesting package. The SE stopping criterion is never met.

Jun 16, 2026

Alex Wainwright

The Item-Writing Problem: How Socially Desirable Items Undermine Scale Validation

Psychometrics

Item Writing

IRT

EFA

Socially desirable attitudinal items produce ceiling effects. Ceiling effects can have deleterious effects on factor analysis. This post traces a plausible pathway from item wording to ceiling effects and then to potential weaknesses in a factor model.

Jun 7, 2026

Alex Wainwright

Good Fit, Fragile Inference

Measurement

SRL

SEM

CFA

An examination of how apparent model fit can mask instability in measurement and parameter estimates.

Dec 27, 2025

Alex Wainwright

Time-on-Task Is Not Self-Regulation: Measurement Validity in Learning Analytics

Measurement

SRL

Learning Analytics

Examining how trace-based SRL measures conflate measurement scale with psychological construct, using a multi-text writing study to illustrate broader problems in the field.

Dec 15, 2025

Alex Wainwright

When Perfect Fit Isn’t Perfect: Rethinking a Trace-Data SEM

SEM

Learning Analytics

This post critically examines a published path analysis linking trace data to self-regulated learning processes. It highlights issues in the model’s specification and parameterisation, and discusses the implications of these problems for interpreting trace-based measures.

Dec 12, 2025

Alex Wainwright

Equating Noise: Re-Examining a GPCM Study Through Simulation

IRT

Simulation

GPCM

This post critically reviews a GPCM-based equating study and uses simulation to assess the stability of its reported parameters. The findings show substantial uncertainty and highlight methodological oversights that undermine the study’s conclusions.

Nov 27, 2025

Alex Wainwright

CAT Isn’t Magic: Why Item Bank Targeting Matters More Than the Algorithm

CAT

IRT

Even the best selection strategy fails when the bank is skewed. This simulation compares linear, random, and 54321 adaptive forms to show where each succeeds, where they break down, and why content coverage drives measurement quality.

Nov 24, 2025

Alex Wainwright

What People Are Really Building in r/AI-Agents

Embeddings

LLMs

Topic Modelling

Using embeddings, STM, and LLM summaries, this post breaks down the dominant themes in r/AI-Agents and examines how discussions evolve across the month.

Nov 18, 2025

Alex Wainwright

Prompt Engineering and Reliability in AI Scoring of Candidate Essays

Psychometrics

AI Scoring

Recruitment

Simulation

This post explores how prompt design and model parameters affect AI-based scoring of candidate essays. Using a simulation study, we test different prompt types, essay strengths, and temperature settings to evaluate consistency, discrimination, and reliability. By the end, you’ll understand the practical trade-offs of using LLMs in recruitment scoring and why human oversight remains essential.

Sep 21, 2025

Alex Wainwright

When Measurement Models Break Down: Revisiting Remote Work and Job Satisfaction Research

EFA

CFA

Measurement

Validation

This post re-examines a remote work study using structural equation modeling and finds that while remote work doesn’t affect job satisfaction, the original survey items measuring ‘job satisfaction’ and ‘teamwork’ actually overlap so much they can’t be distinguished statistically.

Jun 18, 2025

Alex Wainwright

Fixing What’s Broken: Refining a Social Media Use Scale Through Rigorous Analysis

EFA

CFA

Measurement

Validation

Even published measurement scales sometimes need improvement, as this psychometric analysis of a social media use scale clearly demonstrates. What started as a 31-item, four-factor instrument became a more defensible 23-item, three-factor scale through systematic evaluation of item performance and model fit.

Jun 14, 2025

Alex Wainwright

When Small Samples Undermine Measurement: Lessons from Simulating the Overall Disability Scale

IRT

Rasch

Simulation

What happens when robust psychometric tools are used on small samples? Through repeated simulations with the Overall Disability Scale for POEMs, we reveal how even well-designed measures can yield unreliable scores, and what this means for clinicians and researchers relying on patient-reported outcomes.

Jun 8, 2025

Alex Wainwright

Re-Evaluation of the BS Frequency Scale: BS or Not?

Factor Analysis

Scale Development

An exploratory re-analysis of the Bullshitting Frequency Scale using EFA and model diagnostics—questioning its structure, clarity, and construct validity.

Jun 6, 2025

Alex Wainwright

Can GPT Play a Narcissist? Testing Psychometric Consistency Across Fictional Profiles

Item Analysis

Testing whether a GPT model consistently interpret narcissistic traits in fictional character profiles across repeated psychometric assessments, revealing limitations in AI’s ability to detect subtle psychological characteristics.

May 5, 2025

Alex Wainwright

Enhancing Anxiety Assessments: Identifying Key Items and Testing Dimensionality with the Rasch Model

Rasch Modelling

IRT

Item Analysis

Evaluating an anxiety scale for effective item performance and unidimensionality with item analysis and the Rasch model.

Oct 27, 2024

Alex Wainwright

Practical Item Analysis: Understanding Easiness and Discrimination in Cognitive Assessments

Psychometrics

CAT

Simulation

catsim

This post provides a hands-on guide to performing item analysis on cognitive assessment items using Python. Learn how to simulate responses, evaluate item easiness, and calculate discrimination indices. By the end, you’ll be able to assess the quality of your items without relying on complex IRT models, making this approach useful for both item construction and the evaluation of existing item banks. Perfect for practitioners looking to enhance the effectiveness of recruitment assessments or other cognitive tests.

Sep 22, 2024

Alex Wainwright

Sentiment Shifts and Topic Clusters: Dissecting the 2024 Presidential Debate

Text Mining

Sentiment Analysis

Embeddings

Using sentiment analysis and embeddings to explore the 2024 Presidential Debate.

Sep 15, 2024

Alex Wainwright

Transforming Train Performance Data into Insights: Building an R Shiny Dashboard

Dashboard

Performance

Rail

Transforming Overwhelming PDF Performance Data into an Easy-to-Digest Interactive Dashboard.

Jul 11, 2024

Alex Wainwright

Beyond Completion Rates: A Deeper Look into Online Learner Engagement

Learning Analytics

LMS

An example of how you can leverage Learning Analytics to understand engagement with Learning Management System data.

May 26, 2024

Alex Wainwright

Brews and Views: Unveiling the Coffee Habits of YouTube Live Stream Fans

Coffee

Survey

Exploratory Data Analysis

In this exploratory data analysis, we explore coffee preferences and behaviours of individuals who tune into a YouTube Live Stream.

May 19, 2024

Alex Wainwright

Can We Predict Employee Happiness? Exploring Work Enjoyment with Random Forests

Psychometrics

Machine Learning

Predictive Modelling

Using a combination of psychometric data and machine learning, we attempt to predict employee happiness.

May 14, 2024

Alex Wainwright

Simulating the Effects of Item Drift on CATs

Psychometrics

CAT

Simulation

This simulation investigates the impact of item drift on computerised adaptive testing. It demonstrates how item compromise can lead to biased ability estimates and overexposed items, affecting the precision and validity of CATs.

Mar 14, 2024

Alex Wainwright