Running the test, not just writing it

A question bank with a workflow

A question is not a row that exists or does not. It is a draft somebody wrote, a review somebody asked for, a set of verdicts from named reviewers, and an approval. Zayeed models that directly.

• Six states — draft, in review, approved, live, rejected, retired. A paper draws only from live questions; everything else is still being argued about.
• Five roles — reader, author, reviewer, approver, administrator. Which moves you are offered is decided in the engine, not by hiding a button.
• An author cannot approve their own question, and the refusal is a sentence a person can act on rather than a code.
• Wording freezes under review, so a reviewer and an author are never reading two different questions.
• Every move is kept — who moved it, when, from what to what, and the comment they left. That history is what an appeal actually needs.
• Panel verdicts — approve, revise, reject, with the quorum stated. "Approve everything the panel has cleared" acts on the ones that reached it.

A paper built to a specification

The strongest thing a serious awarding body asks for is not a bigger bank. It is a guarantee that every sitting covers the syllabus in the same proportions. A blueprint is that guarantee, written as rows anybody can read aloud.

• So many from here, so many from there — each row draws a stated number of questions matching tags, type, difficulty band, language or text.
• Headroom shown while you write it — what the row asked for, how many live questions the bank can actually supply, and the shortfall if there is one.
• How many different papers it can produce, and which row is limiting that number.
• No double-draw — a question used by one row is not available to the next, so overlapping rows cannot put the same question on the paper twice.
• Deterministic — the same blueprint, bank and seed produce the same paper, months later, at appeal.
• Only live questions are drawn. A blueprint that quietly counted drafts would not be a claim about coverage at all.

A register, groups, and access arrangements

Who is entered, in which group, on what terms — and the variations granted to individuals, each with the reason it was granted.

• Candidates and groups — enter one person or a whole class in a single action.
• Terms of the sitting — when it opens and closes, how long it runs, how many attempts, and whether a password is needed to start.
• Variations resolve, they do not fight — a candidate in two groups gets the most lenient of the terms that reach them, and the screen says which one applied and why.
• Extra time as a percentage, which is how every awarding body writes it, applied as a longer session rather than a clock that runs differently.
• Why a candidate cannot start is a sentence beside their name — not entered, window closed, attempts used, wrong password.
• A variation changes access and nothing else. There is deliberately no field here that can reach the grader, the marking or the pass mark.

Human marking, double marking, moderation

Everything the automatic marker declined to decide arrives in one pile, under one policy that travels inside the .mcq file.

• Rubrics — criteria and levels, each level worth stated marks, with the descriptor visible while you award it.
• One marker or two, independently, with a tolerance that decides what counts as a disagreement rather than an argument about it afterwards.
• Blind second marking — the second marker is not shown the first mark, and that is enforced where the marks are read, not by a screen that hides a number it was handed.
• Anonymous marking — candidate names hidden while marking, on request.
• A moderator settles it, and their mark outranks both. Nothing is overwritten: a revised mark is a new record, so the original never quietly disappears.
• Marker reliability — how many each marker has done, their mean, their bias against the panel, and their disagreements. A marker is never shown their own bias mid-pile, because knowing it changes how they mark the rest.
• Deal the pile out deterministically across a marking team.

Where the pass mark came from

Most assessment sets a pass mark by tradition — sixty per cent, because it has always been sixty per cent. That is fine for a class test and indefensible for anything that decides a career.

• Angoff and modified Angoff — each judge states, per question, how likely a just-barely-qualified candidate is to answer it correctly.
• Round one is judged blind. A judge who can see the panel mean before committing is agreeing, not judging. The numbers open up between rounds.
• The evidence, not just the number — the cut score, the spread between judges, the standard error, the questions the panel disagreed about, and the judges sitting furthest from it.
• An outlier is a person to ask, not a number to average away, and the page says so.
• Moving the cut by standard errors is offered explicitly, with the sentence explaining what it did.
• Accepting the standard writes the pass mark onto the paper with the evidence attached, so nobody has to remember where 62% came from.

Sections, with their own clocks and their own doors

A real paper is rarely one undifferentiated list of questions. It has a listening section that runs for ten minutes and then stops; a reading section you may work through in any order; a recall section you cannot go back into once you have left it. A set becomes a section as soon as it carries one of three rules — and a paper whose sets carry none of them behaves exactly as it always did.

• Its own clock, counted only while the candidate is in that section and the sitting is running. A supervised rest break does not spend it.
• Walking away cannot outlast it. Time spent away is capped at the limit and the section closes — an absence cannot buy time in a section it did not spend.
• Sealed sections close behind the candidate, even on a paper that otherwise allows going back. For where a later question gives away an earlier one.
• Back navigation per section, judged by the section being entered rather than the one being left.
• When time runs out the candidate is carried on to the next open section, and the move is written into the log as a real navigation — an attempt that jumped sections with nothing to show for it could not explain itself at an appeal.
• A late answer is refused in words. A refusal a candidate cannot read is indistinguishable from a broken button, which under exam conditions is the difference between an incident and a candidate carrying on.
• Shuffling stays inside a section, and sections stay in the order they were written — so a reading question cannot be shuffled into the listening paper.
• The rules travel in the file. The same questions under different section clocks are a different examination, and the .mcq pack says which one it is.

The same pack, played in front of a room

Not every assessment is a sitting. Sometimes the paper is a warm-up in a lecture theatre, a check for understanding in period four, or a show of hands at a conference — and for that, the friction of accounts and installs is the whole problem. A pack you already own can be opened as a live room: the host gets a six-character code, everyone else joins from the phone already in their hand.

• No account and no install to join. A phone needs the code and a browser. Nothing is downloaded and nothing is signed up for.
• A poll or a quiz. A poll marks nothing and keeps no score. A quiz is marked by the same engine a formal sitting uses, so the score agrees with the paper rather than approximating it.
• The projector cannot show who voted for what. Not as a setting — the tally has nowhere to put a participant identifier, so there is no relay change and no interface bug that could put a name on the wall.
• Closing and revealing are two separate actions. If the bars appeared the instant answering stopped, the last person to vote would watch their own answer move the chart. In a room of three that identifies them.
• An anonymous room never asks for a name, so there is none to leak afterwards.
• Phones do not see the running count by default. Someone watching the bars move is being told the popular answer before they commit to their own.
• A quiz refuses late arrivals, because a candidate who answered four of eight questions has not scored lower — they have scored something that cannot be compared. A poll lets them wander in.
• Reloading the big screen does not lose the room. The room is an event log replayed by the same kind of reducer the rest of the product uses, so there was never any state living only in that browser tab.
• A question typed mid-room is a real question. The host improvises one, and it is an item of a real type marked by the same engine — so the room can be saved afterwards as a .mcq pack and sat properly later.
• Ratings, net promoter and rankings, none of which are new question types: a rating is a numeric item carrying its scale, a ranking is the ordering type. That is why all four shells got them at once.
• Questions from the floor, moderated and upvoted — and who upvoted what never leaves the room.

All of it, on every shell

These are controllers in the shared core, not screens in one app. The web build, the Windows and Linux desktop and the Android build each render the same review queue, blueprint table, register, marking pile and standard-setting panel, against the same local database — offline, and with the same refusals in the same words.