Running the test, not just writing it

A question bank with a workflow

A question is not a row that exists or does not. It is a draft somebody wrote, a review somebody asked for, a set of verdicts from named reviewers, and an approval. Zayeed models that directly.

• Six states — draft, in review, approved, live, rejected, retired. A paper draws only from live questions; everything else is still being argued about.
• Five roles — reader, author, reviewer, approver, administrator. Which moves you are offered is decided in the engine, not by hiding a button.
• An author cannot approve their own question, and the refusal is a sentence a person can act on rather than a code.
• Wording freezes under review, so a reviewer and an author are never reading two different questions.
• Every move is kept — who moved it, when, from what to what, and the comment they left. That history is what an appeal actually needs.
• Panel verdicts — approve, revise, reject, with the quorum stated. "Approve everything the panel has cleared" acts on the ones that reached it.
• Every wording is kept — each edit stores the version it replaced, numbered, with the parts that changed named in an author's words: the question, the options, the correct answer, the marks. A question whose correct answer moved between two sittings is a different thing from one whose typo was fixed, and only the first explains why a cohort did worse.
• Opening and saving is not an edit. A version is recorded only when something actually changed — a history where half the entries say nothing happened is one nobody reads, and therefore no history at all.
• Restoring does not erase. Putting an old wording back writes a new version on top rather than removing what came between, so the bank can still answer what candidates actually sat that term.

A paper built to a specification

The strongest thing a serious awarding body asks for is not a bigger bank. It is a guarantee that every sitting covers the syllabus in the same proportions. A blueprint is that guarantee, written as rows anybody can read aloud.

• So many from here, so many from there — each row draws a stated number of questions matching tags, type, difficulty band, language or text.
• Headroom shown while you write it — what the row asked for, how many live questions the bank can actually supply, and the shortfall if there is one.
• How many different papers it can produce, and which row is limiting that number.
• No double-draw — a question used by one row is not available to the next, so overlapping rows cannot put the same question on the paper twice.
• Deterministic — the same blueprint, bank and seed produce the same paper, months later, at appeal.
• Only live questions are drawn. A blueprint that quietly counted drafts would not be a claim about coverage at all.

A register, groups, and access arrangements

Who is entered, in which group, on what terms — and the variations granted to individuals, each with the reason it was granted.

• Candidates and groups — enter one person or a whole class in a single action.
• Terms of the sitting — when it opens and closes, how long it runs, how many attempts, and whether a password is needed to start.
• Variations resolve, they do not fight — a candidate in two groups gets the most lenient of the terms that reach them, and the screen says which one applied and why.
• Extra time as a percentage, which is how every awarding body writes it, applied as a longer session rather than a clock that runs differently.
• Why a candidate cannot start is a sentence beside their name — not entered, window closed, attempts used, wrong password.
• A variation changes access and nothing else. There is deliberately no field here that can reach the grader, the marking or the pass mark.

Booking the seat

Everything above happens after somebody has a desk. A window says when a paper may be sat and in which rooms, a venue says how many desks are in one of those rooms, and an appointment is one candidate in one of those desks at one time. The candidate books it themselves, or the examinations office books it for them, and both go through the same rules.

• Seats taken are counted, never cached. Occupancy is worked out from the appointments that exist, by overlapping intervals, every time somebody asks. A stored count is a double-booking waiting for two people to press the button at once — and a candidate with 25% extra time holds a desk into the next slot, which a per-slot counter cannot see at all.
• Arrangements are applied when the slot is offered, not when the sitting starts, so a candidate is never shown a start that cannot fit the sitting they are entitled to.
• Reserved desks exist and are left empty on purpose — a separate room, an adjustable desk, a station with a reader — held back from the general queue for the candidates whose arrangements need them.
• Every refusal is a sentence, and the survivable ones carry alternatives. Not enrolled, booking not open, booking closed, too late to move, the room is full, it does not fit: each one names the next starts this candidate could take instead. A refusal with nowhere to go is a telephone call.
• The candidate portal shows one next action. When they sit, where it is, how to get in, what to bring, the checklist — and exactly one thing to do next, in words. A page that offers somebody four tasks the night before an examination gets none of them done.
• The portal is on all four — the browser, Windows, Linux and Android, from one shared controller rather than four implementations, so no two of them can disagree about whether a seat may still be moved. Reading the timetable works with no network and says when what is on screen was last true; booking, moving and giving up a seat need the server, and an application that cannot reach it withholds those controls and says why rather than queueing a promise about a desk it cannot see.
• Two institutions stay two institutions. Somebody sitting at a school and a college has a candidate record at each, with different numbers and different arrangements, and the portal shows two cards rather than merging them into one wrong one.
• An admission records who admitted them. Not "the system" — a named person, because that is the row an appeal about an identity check turns on.
• The office sees what turns into telephone calls: seats, booked, sat, missed, cancelled, and how many are still unbooked with booking about to close.
• All of it over the API. Twenty-five scheduling operations on /v1, so an institution provisions its people, books a sitting, runs it proctored and pulls the results without opening a support ticket.

Telling the candidate

A sitting generates the notices it needs — booked, moved, cancelled, a reminder, an outstanding checklist, booking closing, nothing booked, a result released — each with a stable identifier, so whatever sends it can refuse to send it twice without keeping a ledger of its own. What happens to them next is a decision, and it is yours.

• We are not the sender, ever. A notice leaves through your own SMTP relay, or through a URL on your own side that receives it signed and does whatever you do with it, or through nothing at all — which is the default on a fresh installation and a real answer rather than an unfinished one. Zayeed has no address, no sending domain and no mail reputation of its own to lend to your examination timetable.
• An address has to have somebody behind it. Every address carries where it came from — typed, imported, from a directory, from the candidate — and a transport can require that a named person confirmed it before anything is written to it. Until then the notice is held, with the reason and the next action in words beside the candidate’s name. Not failed, not dropped, and not quietly sent: the address a spreadsheet carried in last term is exactly the one software should refuse to mail an examination time to.
• Nobody is told twice. The outbox is keyed on the notice’s own identifier, so queueing the same notice twice is the same row twice, and a dispatcher that died between the send and the ledger write cannot say it again.
• A refusal is read, not retried blindly. A temporary refusal comes back on a stated backoff; a relay that keeps refusing is disabled rather than hammered, and says why. A permanent refusal suppresses that address instead of being tried at it again, and the suppression is visible with its reason rather than looking like an address nobody ever had.
• Nothing sweeps your schedule on its own. The server sends what somebody queued, from the management screen or from a request. “The upgrade mailed our candidates” is not a sentence this product can produce.
• The relay credential is written and never read back, stays on the machine it was configured on, is never synced to a phone, and is not reachable by an API token — a scoped token that could redirect an institution’s outbound notices is the same failure as a token that can mint a token.

Human marking, double marking, moderation

Everything the automatic marker declined to decide arrives in one pile, under one policy that travels inside the .mcq file.

• Rubrics — criteria and levels, each level worth stated marks, with the descriptor visible while you award it.
• One marker or two, independently, with a tolerance that decides what counts as a disagreement rather than an argument about it afterwards.
• Blind second marking — the second marker is not shown the first mark, and that is enforced where the marks are read, not by a screen that hides a number it was handed.
• Anonymous marking — candidate names hidden while marking, on request.
• A moderator settles it, and their mark outranks both. Nothing is overwritten: a revised mark is a new record, so the original never quietly disappears.
• Marker reliability — how many each marker has done, their mean, their bias against the panel, and their disagreements. A marker is never shown their own bias mid-pile, because knowing it changes how they mark the rest.
• Deal the pile out deterministically across a marking team.

Where the pass mark came from

Most assessment sets a pass mark by tradition — sixty per cent, because it has always been sixty per cent. That is fine for a class test and indefensible for anything that decides a career.

• Angoff and modified Angoff — each judge states, per question, how likely a just-barely-qualified candidate is to answer it correctly.
• Round one is judged blind. A judge who can see the panel mean before committing is agreeing, not judging. The numbers open up between rounds.
• The evidence, not just the number — the cut score, the spread between judges, the standard error, the questions the panel disagreed about, and the judges sitting furthest from it.
• An outlier is a person to ask, not a number to average away, and the page says so.
• Moving the cut by standard errors is offered explicitly, with the sentence explaining what it did.
• Accepting the standard writes the pass mark onto the paper with the evidence attached, so nobody has to remember where 62% came from.

Two versions of the paper, one scale

Any paper that runs more than once runs in more than one version, and the versions are never quite equally hard. Without equating, a candidate's result depends partly on which morning they sat — which is exactly the unfairness a pass mark is supposed to be immune to.

• Four methods — mean/mean, mean/sigma, Haebara and Stocking–Lord, each with a sentence saying when it is the right one.
• The anchor is found, not nominated. The common questions are whatever genuinely appears on both papers. A hand-picked anchor is how a link gets quietly tuned until it gives the answer somebody wanted.
• It refuses a thin anchor — fewer than three common questions and there is no defensible link to compute — and it refuses a form that has not been sat thirty times, because an equating between two uncertain calibrations is a transformation between two sets of noise with a slope printed beside it.
• Drifted anchors are named and set aside. A question that has become easier between sittings is most often a question that has escaped. That is a security finding before it is a statistical one.
• Shown as what it does to a person — a measure at three points on the scale, before and after, with its standard error. A slope and an intercept are not readable as an effect on a candidate.
• A link is never edited. Each one is stored as its own row: a constant that changed after results were issued is a different constant, not a corrected one, and the candidates scaled under the two versions are not comparable.
• Nothing is rescaled automatically. The link is an artefact to review and then decide to operate from. Software that applies one the moment it is computed has taken that decision away from the people whose name is on the certificate.

Sections, with their own clocks and their own doors

A real paper is rarely one undifferentiated list of questions. It has a listening section that runs for ten minutes and then stops; a reading section you may work through in any order; a recall section you cannot go back into once you have left it. A set becomes a section as soon as it carries one of three rules — and a paper whose sets carry none of them behaves exactly as it always did.

• Its own clock, counted only while the candidate is in that section and the sitting is running. A supervised rest break does not spend it.
• Walking away cannot outlast it. Time spent away is capped at the limit and the section closes — an absence cannot buy time in a section it did not spend.
• Sealed sections close behind the candidate, even on a paper that otherwise allows going back. For where a later question gives away an earlier one.
• Back navigation per section, judged by the section being entered rather than the one being left.
• When time runs out the candidate is carried on to the next open section, and the move is written into the log as a real navigation — an attempt that jumped sections with nothing to show for it could not explain itself at an appeal.
• A late answer is refused in words. A refusal a candidate cannot read is indistinguishable from a broken button, which under exam conditions is the difference between an incident and a candidate carrying on.
• Shuffling stays inside a section, and sections stay in the order they were written — so a reading question cannot be shuffled into the listening paper.
• The rules travel in the file. The same questions under different section clocks are a different examination, and the .mcq pack says which one it is.

The same pack, played in front of a room

Not every assessment is a sitting. Sometimes the paper is a warm-up in a lecture theatre, a check for understanding in period four, or a show of hands at a conference — and for that, the friction of accounts and installs is the whole problem. A pack you already own can be opened as a live room: the host gets a six-character code, everyone else joins from the phone already in their hand.

• No account and no install to join. A phone needs the code and a browser. Nothing is downloaded and nothing is signed up for.
• A poll or a quiz. A poll marks nothing and keeps no score. A quiz is marked by the same engine a formal sitting uses, so the score agrees with the paper rather than approximating it.
• The projector cannot show who voted for what. Not as a setting — the tally has nowhere to put a participant identifier, so there is no relay change and no interface bug that could put a name on the wall.
• Closing and revealing are two separate actions. If the bars appeared the instant answering stopped, the last person to vote would watch their own answer move the chart. In a room of three that identifies them.
• An anonymous room never asks for a name, so there is none to leak afterwards.
• Phones do not see the running count by default. Someone watching the bars move is being told the popular answer before they commit to their own.
• A quiz refuses late arrivals, because a candidate who answered four of eight questions has not scored lower — they have scored something that cannot be compared. A poll lets them wander in.
• Reloading the big screen does not lose the room. The room is an event log replayed by the same kind of reducer the rest of the product uses, so there was never any state living only in that browser tab.
• A question typed mid-room is a real question. The host improvises one, and it is an item of a real type marked by the same engine — so the room can be saved afterwards as a .mcq pack and sat properly later.
• Ratings, net promoter and rankings, none of which are new question types: a rating is a numeric item carrying its scale, a ranking is the ordering type. That is why all four shells got them at once.
• Questions from the floor, moderated and upvoted — and who upvoted what never leaves the room.

All of it, on every shell

These are controllers in the shared core, not screens in one app. The web build, the Windows and Linux application and the Android build each render the same review queue, blueprint table, register, marking pile and standard-setting panel, against the same local database — offline, and with the same refusals in the same words.