Command: cat projects/hiring-challenges.md
A factory of technical hiring challenges, with a live defense and a human decision
An internal tool that creates technical hiring challenges for 22 tracks at 4 levels, from intern to senior. Each challenge comes with a brief, a starter kit, an answer key, a rubric based on observable behavior, a live defense script and hidden tests, and it can only be published after showing that a language model alone cannot solve it. Evaluation is pseudonymized, the defense weighs at least half of the score and a person always makes the decision. As of September 2026, no real candidate had been evaluated with it.
open the demoa simulation of the system with fictional data · 6 screens
- Role
- Author of the method and the tool
- Period
- Sep 2026 – Sep 2026
- What I did
Me: The evaluation method, the rubrics per level, the quality gates, the ethics and LGPD rules, the command line tool, the privacy barrier around external models, and the catalog, packaging and data purge scripts.
Private project, described without identifying the client, product or company.
Diagram
Everything runs in a local tool. The challenge sheets, the method and the rubrics live in a note vault; language models generate the artifacts, and gates with evidence and a date decide publication, under the rule that whoever creates does not approve. A privacy barrier lets only candidate material out to the external models. The package sent to the candidate holds only the candidate's folder, checked by the confidentiality guard. The delivery comes back pseudonymized, the evaluators score on their own, there is a live defense, and the report supports a decision that is always made by a person.
Context
A business group needed to structure and hire its technology team. The classic technical test, an isolated take-home, no longer tells apart someone who understands from someone who only delivers: a language model with code execution solves it in minutes. I wanted a process that measured how a person thinks and decides, without watching their machine and without asking for free work.
What I built
- The method. Twelve mechanisms that make an AI-only solution insufficient by design. Rubrics based on observable behavior across eight dimensions, with weights per level, knockout criteria and an authorship lock: with an authorship score of 2 or lower, the opinion does not recommend the person, not even for the level below.
- The creation pipeline. Gates G0 to G5 with evidence and a date, numbered human action items and a catalog generated from the challenge sheets. Artifact generation uses language models under fixed rules: whoever creates does not approve, and the reviews receive the finished package.
- Hardening. Only candidate material goes to at least three models, with neutral file names. Each answer runs against the trap and the hidden suite and gets two scores, submission only and submission plus defense. A challenge is only published with no vulnerable result and at most one partial result covered by the live part.
- Sending and evaluation. A script packages only the candidate folder, runs the confidentiality guard on the copy and blocks the package if an item is still open. During evaluation, the submission is pseudonymized before anyone reads it, evaluators score alone and with evidence, and a two-point gap requires a discussion with new evidence.
Ethics and LGPD in the design
- Human decision. The opinion supports the decision and says so in a mandatory sentence; the hiring manager decides (LGPD, Brazil’s data protection law, article 20).
- Pseudonyms. Each candidate becomes a code before anyone reads the submission, and the table linking code and name lives outside the repository.
- A deletion date. Submissions and notes are purged by script after 6 months, or 12 with consent for the talent pool.
- No free work. Beyond 4 hours of asynchronous work, the challenge is paid. No paid hardware or software is required, and accommodations are offered before the candidate accepts.
- Criteria. No discriminatory criteria (Brazilian Law 9,029/95), and fairness is one of the reviews before publishing.
- Transparency. Candidates know the AI-use policy before they start and declare in their decision log where they used it.
Status
It is an internal tool, local and never deployed. The catalog has five challenges, four drafts and one ready to publish, waiting on items only people can resolve. No challenge has been used, no real candidate has been evaluated and no hire has come out of it as of September 2026. The evaluation screens in the demo show the templates filled in with a fictional candidate.
What I would do differently
I would run the first challenge with a real person before writing the fifth. All the calibration so far comes from simulated candidates and from models, and only a real run will show whether sixty minutes of defense are enough to assess a mid-level engineer.
Constraints
- A language model solves an isolated take-home in minutes; the process has to measure what the person understands and decides.
- No surveillance software on the candidate's machine.
- Panel material and identified submissions never leave for an external service.
- Always a human decision, no discriminatory criteria, and personal data with a deletion date.
- No free work beyond 4 hours of asynchronous effort.
Decisions
The live defense weighs at least half of the score
- Context
- In the first review cycle, an isolated take-home was solved by a language model with code execution, with a simulated score above the hire band from the submission alone.
- Choice
- The final score is 40% of the lower of submission and defense plus 60% of the defense. If any model's submission-only score goes above 3.0 during hardening, the live weight stays at 50% or more and authorship becomes a knockout criterion.
- Gains
- A good submission without understanding does not pass.
- The defense measures what the person decides and can explain, not only the code they hand in.
- Costs
- Each candidate costs an hour of a live panel, which limits how far the process scales.
Against AI-only solutions by design, not by surveillance
- Context
- Monitoring the candidate's screen or camera is invasive and easy to get around, and banning AI does not reflect real work.
- Choice
- Twelve design mechanisms, among them a plausible trap, staged reveals, "explain this line", an injected failure, a planted pull request and hidden tests. At least 7 per challenge, and 9 for mid-level and senior. Candidates may use AI in the take-home and declare where they used it.
- Gains
- No monitoring software on anyone's machine.
- The AI-use policy is known before the candidate accepts.
- Costs
- Every challenge becomes more expensive to design and to calibrate.
Whoever creates does not approve
- Context
- Whoever wrote a challenge cannot see its gaps.
- Choice
- Gates G0 to G5 with evidence and a date. Artifact generation uses language models, and the hardening, simulated candidate, fairness, confidentiality and final critique reviews receive the finished package, without the author's reasoning. Nothing is published while a human action item is open.
- Gains
- Gaps in a brief show up before any candidate sees it.
- Every approval leaves evidence that can be checked later.
- Costs
- A challenge takes several cycles; none of the four in the first cycle passed the design review without changes.
A privacy barrier around external models
- Context
- Hardening sends the challenge to several language models, and the answer key, the knowledge base and identified submissions must not leave the machine.
- Choice
- Only candidate material goes out, with neutral file names. Everything else is refused before the call, each process gets only its own credential, paths are confined to the project folder and the text goes through standard input, with fixed arguments.
- Gains
- An operating mistake does not send the answer key out.
- Costs
- The barrier has to keep up with every new data source.
- A real finding showed that an attachment's path gave the answer away, so only the file's base name is exposed now.
A pseudonym before anyone reads, and a purge date
- Context
- A name, age or school in a file sways whoever evaluates it, and candidate data cannot be kept forever.
- Choice
- A code per candidate, the author removed from the submission's git metadata, an identity table outside the repository, an opinion with the mandatory human-decision sentence and a scripted purge after 6 months, or 12 with consent for the talent pool.
- Gains
- Evaluators do not know who the candidate is.
- The data has an expiry date, and the date is in the opinion.
- Costs
- Feedback requires re-identifying through the external table, one more manual step.
Stack and why
- Node.js scripts with no dependencies
- Catalog, gate status, validation, packaging and purge run on the command line.
- A Markdown note vault (Obsidian)
- Method, role matrix, challenge sheets, evaluation templates and the generated catalog, readable by people who do not code.
- Mermaid
- Flows and diagrams versioned with the notes.
- CLI and MCP server for language models
- Hardening against several models, with a privacy barrier and confined paths.
- A local Node panel
- Follow the pipeline through an event stream, on localhost only.
Results
A matrix of 22 tracks at 4 levels (88 roles), with general anchors, format and maximum duration per level.
audited private repositoryThe project's role matrix(Audit of the private repository, Sep 2026)12 mechanisms against AI-only solutions, with a minimum of 7 per challenge and 9 for mid-level and senior.
audited private repositoryThe project's method notes(Audit of the private repository, Sep 2026)A catalog of 5 challenges: 4 drafts and 1 ready to publish, waiting on human action items such as a real contact channel. None published, no real candidate evaluated and no hire made with the tool as of September 2026.
audited private repositoryGenerated catalog and challenge sheets(Audit of the private repository, Sep 2026)In the first cycle, an isolated take-home solved by a language model had a simulated score of 3.3 to 3.8 from the submission alone, which fell to about 2.45 with the defense; the publishing rule changed because of it.
self-reportedThe project's lessons log
Security angle
Attack surface
- Panel material (answer key, rubric and hidden tests) that must reach neither the candidate nor an external service.
- Candidate submissions, with personal data.
- Calls to external language models from the command line.
- The package sent to the candidate.
- The local event panel.
Controls in place
- A privacy barrier that refuses panel material, the knowledge base and non-pseudonymized submissions before the call.
- Confined paths that refuse .., another drive, network paths and junctions pointing outside the folder.
- A separate credential per process, fixed arguments and text through standard input.
- A confidentiality guard that looks for secrets, personal data and panel excerpts in the package, without printing the values.
- Packaging that copies only the candidate folder, blocks on open publishing items and names the archive with a neutral code.
- Pseudonymized submissions, including the author in git metadata, and a scripted purge.
- A panel on localhost only, with an origin check, a body size cap and log rotation.
What I would test today
- An attachment with a symbolic link pointing at the panel material.
- Hidden instructions in the brief or the submission, trying to make a model read other files.
- Metadata of documents and images inside the package sent to the candidate.
Evidence
- audited private repositoryAudited private folder (local tool, never deployed)(Audit of the private repository, Sep 2026)