Command: cat projects/content-pipeline.md
A question pipeline with style measurement and an acceptance gate
My own command-line tools and scheduled routines that measure how an exam board writes, check every batch of questions against that measurement and deliver only drafts to the study platform's panel, where a person approves or discards them. Quality is checked with numbers and exit codes, and the gate itself is attacked by tests. Internal use, with no business results to show.
open the demoa simulation of the system with fictional data · 5 screens
- Role
- Author of the tools and of the operating contract
- Period
- Aug 2026 – present
- What I did
Me: The pipeline design, the style measurements, the acceptance gate's criteria, the exit-code contract, the tools, the adversarial tests and the idempotent delivery to the panel.
Private project, described without identifying the client, product or company.
Diagram
Public past exams, with their hash checked on download, give the exam board's measured style profile. A scheduled routine drafts the day's batch from that profile, and the acceptance gate checks integrity, repetition, style and syllabus coverage, with a verdict and an exit code as the contract. In CI, an adversarial generator attacks the gate. An accepted package goes to the ingestion API with an idempotency key and a receipt, and reaches the panel as an inactive draft until human review.
Context
A question catalog grows at the speed of whoever reads the exam notice and past exams and writes in the board’s style. A routine that drafts questions with a language model speeds up the writing, but it leaves predictable marks: the correct option tends to be the longest, topics repeat, and a routine that fails usually fails silently. This pipeline measures those marks before anything reaches the panel of the study platform.
What I built
- Measuring the board’s style. Tools that download public past exams with a checked hash, split the questions, match each one to the final answer key and measure how the board writes.
- Acceptance gate. Every package goes through integrity, repetition against the archive and earlier packages (statement plus options), style against the measured profile and coverage of the exam notice, and comes out with a verdict and an exit code.
- Routines with a contract. One routine drafts the day’s batch from the measured profile; others check that the routines ran and measure the product from the outside. They all only read and report, and silence is a result too.
- Idempotent delivery. An accepted package goes to the panel through the ingestion API with an idempotency key and a receipt, and it arrives as a draft: inactive questions, no price and no official mock exam.
- Generated panel. A note generated from the log tells what was not measured apart from what never ran, and points out diverging instructions.
Challenges
- A lazy adversary. The first version of the adversarial generator’s good case produced questions from a single template, and the gate rejected the batch for 435 repeated pairs. The gate was right, and the test was rewritten.
- A test that tested nothing. The mutation test showed that one of the gate’s limits could be switched off without failing any test. It became a new case for the adversary.
- A swapped exit code. A corrupted file returned 1 (found a problem) when it should have returned 2 (could not look).
- A closed network in the cloud. Reading official pages failed on every domain tested. The evidence contract was rewritten in levels, and monitoring the product moved to the local machine.
Status
Internal use, with no production results to show. At the last measurement, the panel flagged routines whose published instructions differed from the repository version, which is exactly what the detector is there to find.
Constraints
- Nothing reaches a student without human review, so the pipeline only delivers drafts.
- Scheduled routines run with nobody watching, and a silent failure has to become a signal.
- Tools with no third-party dependencies, so they run on any machine with Node.
- The archive of past exams stays on the local machine.
Decisions
Style measured, not described
- Context
- Asking for questions in an exam board's style produces plausible text with predictable marks, such as the correct option almost always being the longest one, which lets a student get it right by length.
- Choice
- Measure the board's profile on past exams (statement and option length, negative commands, Roman-numeral items, the answer distribution and how often the correct option is the longest) and use the same numbers in the drafting instructions and in the acceptance gate.
- Gains
- Style becomes something you can check, package by package.
- An answer given away by its length is stopped before it reaches the panel.
- Costs
- It depends on a local archive of past exams and on measuring again for every board.
Exit codes as a contract
- Context
- A tool that swallows an error looks the same as one that found nothing, and routines decide their next step by the exit code.
- Choice
- Every tool follows the same contract. Zero means it measured and found nothing, one that it measured and found something that needs a look, and two that it could not measure.
- Gains
- Not being able to look no longer passes for everything being fine.
- Scripts chain decisions without parsing text.
- Costs
- Each tool has to tell a read failure from a finding, and one case got it backwards and was fixed.
Attacking my own gate
- Context
- A gate that nobody tries to get past may be approving things by accident.
- Choice
- An adversarial generator builds packages designed to get through the gate, and none may pass. A mutation test switches a check off on purpose and verifies that the adversary notices.
- Gains
- It found a limit that could be switched off without failing a single test.
- Costs
- More test code to maintain, and the adversary needs reviewing too.
Routines read and report, and only drafts enter the product
- Context
- Scheduled routines run unsupervised, and a routine with write access could publish a mistake to every student.
- Choice
- Routines only read and send a report by e-mail. The only write to the product is sending a package through the ingestion API, with an idempotency key and a receipt, and it arrives as a draft pending review.
- Gains
- A bad run publishes nothing.
- Resending the same package does not duplicate content.
- Costs
- Throughput is limited by human review, and receiving each package is still a manual step.
Versioned instructions with a drift detector
- Context
- The instructions running in the cloud can change without the repository's file changing, and nobody notices until the output gets worse.
- Choice
- Every set of instructions is a versioned file, with the published copy kept next to it, and a detector compares the two.
- Gains
- Drift shows up on the panel instead of as a surprise in a report.
- Costs
- Every change means publishing and checking again.
Stack and why
- Plain Node.js (ESM)
- No third-party dependencies, with a PDF reader and a spreadsheet writer of my own.
- node:test
- Regression tests, the adversarial generator and the mutation test.
- GitHub Actions
- Runs the suite, checks the derived instructions and attacks the gate on every push.
- Scheduled routines and a local task
- Batch production runs in the cloud; watching the product runs on the machine.
- Obsidian
- A panel generated from the measured log, which nobody edits by hand.
Results
About 11.5k lines across 50 tool files, with no third-party dependencies.
audited private repositoryRepository count(Audit of the private repository, Sep 2026)CI that runs the regression tests, the full suite, the derived-instructions check and the adversarial generator against the gate.
audited private repositoryThe repository's workflow(Audit of the private repository, Sep 2026)117 tests in 14 sections, and a similarity limit of 0.75 to flag repeats, against 0.211 for the most similar legitimate pair measured so far.
self-reportedThe project's documentation
Security angle
Attack surface
- Downloading public past exams from official pages.
- Question packages received by e-mail and read from disk.
- The panel's ingestion API, with a machine credential.
Controls in place
- Fetching with an explicit origin, pinned DNS and public addresses only, against SSRF.
- A hash checked on every downloaded exam and every received package.
- Package reads with a size limit, refusing symbolic links and junctions.
- Delivery with an idempotency key and a lookup before resending, and the panel only receives drafts.
- Scheduled routines with no write access to the product at all.
What I would test today
- A package with malicious HTML or a link in a statement, from the gate to the panel's review screen.
- An official page that redirects to an internal address during collection.
- The same package resent with a changed body and the same idempotency key.
Evidence
- audited private repositoryAudited private repository(Audit of the private repository, Sep 2026)