Distractor generation with Hugging Face models

maze-distractor-generator picks a distractor for every word of your materials. It replaces the original distractor generation code (maze_automate) described in Boyce et al. (2020) and Boyce & Levy (2023). The main differences:

It works with current language models: any Hugging Face causal model (GPT-2, Pythia, Llama, …), masked models (BERT, RoBERTa), or API models via liteLLM.
There is a review workflow: mark bad distractors and regenerate just those.
The default word list is curated to avoid offensive and sensitive words.
It has JSON output for jsPsych.

The idea is the same as before (see What is A-maze?). It is set up for English; see A-maze in other languages for adapting it. For each word, draw candidate words of similar length and frequency, and keep one the language model finds very surprising in that position.

Install

git clone https://github.com/vboyce/maze-distractor-generator
cd maze-distractor-generator
python -m venv .venv
.venv/bin/pip install -r requirements.txt

Without an NVIDIA GPU, install the CPU build of torch first:

pip install torch --index-url https://download.pytorch.org/whl/cpu

Materials

A CSV with a header row: type (condition; passed through), item_num, sentence, and optionally labels.

type,item_num,sentence,labels
sub_rel,3,The cat who the dog scared hid in a box.,pre_1 pre_2 who art noun verb main_verb post_1 post_2 post_3
obj_rel,3,The dog who scared the cat sniffed around the couch.,pre_1 pre_2 who verb art noun main_verb post_1 post_2 post_3
filler,4,Birds sing in the morning.,

Sentences with the same item_num share distractors. Without labels they are matched by word position; with labels, words with the same label get the same distractor. In the example, the relative-clause nouns get the same distractor even though they are in different positions. This matters for minimal pairs: if the distractors differ between conditions, so might the RTs.

Running it

# CSV output
.venv/bin/python distract.py materials.csv distractors.csv -p params.txt

# JavaScript module for jsPsych (items have item_type, id, sent, distractor, labels)
.venv/bin/python distract.py materials.csv stimuli.js -p params.txt --format json

params.txt sets the model and thresholds. The key settings are:

min_delta and min_abs: a distractor must be at least min_delta bits more surprising than the real word, and at least min_abs bits surprising.
num_to_test: how many candidates to try per position.
model and backend.
max_repeat: how many times any word may be used as a distractor.

The trade-offs between them are discussed in Parameter considerations; min_delta, min_abs and num_to_test mean the same as in the original code. The README has the full table. An unknown key in the parameters file is an error.

Reviewing distractors

Some automatic distractors will be plausible continuations, or words you don’t want participants to see. This matters most for critical regions and for special populations such as children. The review workflow:

Generate with a review file, listing a few alternatives per position:

.venv/bin/python distract.py materials.csv distractors.csv -p params.txt --longform review.csv --num-options 3

Open review.csv (one row per word position). Mark bad distractors in the rejected column.

Regenerate only those:

.venv/bin/python distract.py materials.csv stimuli.js --format json -p params.txt \
    --rejection-file review.csv --longform review_2.csv

Everything not rejected is kept. stimuli.js has the final distractors for every sentence. Repeat with review_2.csv as needed.

Other checks: pilot the materials and regenerate distractors that several participants get wrong. check_distractors.py, which asks an LLM whether each distractor is a grammatical continuation, is experimental. In a test run it judged 8% of real words ungrammatical, so it isn’t reliable yet.

Which model?

Small causal models work well and run quickly on a laptop; we have used gpt2, distilgpt2 and EleutherAI/pythia-160m. benchmark.py compares the run time of several models on the same input. Larger models give better surprisal estimates but are slower. Because surprisal comes from subword tokens, words split into many tokens get higher surprisal; that’s one reason the targets are thresholds rather than exact values.