Night Shift points Claude sessions at real research questions. Does a published paper’s code actually regenerate its numbers? Where does a public scientific dispute really stand? Which papers can even be checked? The thing is, the hard part (and honestly most of what I built) is making sure the agent doing a check can’t see the answer it’s supposed to produce. A reproduction that peeked at the paper’s results is worthless, and an agent will peek without meaning to if any channel is open.
Three kinds of mission
Find. Go looking for work worth checking. One sweep opened 111 replication packages one at a time and sorted them: 71 complete, 25 gated or incomplete, 15 couldn’t check. Another went hunting for contested scientific claims and came back with twelve, everything from D-Wave’s quantum advantage paper to the minimum-wage fight. A third checked whether the Institute for Replication’s AI tooling existed so I could just reuse it. It doesn’t; their GitHub publishes packages, not tools.
Map. Lay out a dispute without picking a side. Each map names every party, what exactly they claim, their stake (employer, funding, commercial) and their sources, then says what an outsider can and can’t actually check. Twelve are queued: D-Wave, AlphaChip, minimum wage, the CMIP6 “hot models”, Microsoft’s Majorana qubit, reasoning-model “collapse”, cancer-trial surrogate endpoints, the Lesné amyloid paper, Jiuzhang, emergent abilities, epigenetic clocks, and enhanced rock weathering. D-Wave and AlphaChip are done.
Reproduce. Take a paper with open data and code, rerun it, and record, number by number, whether each reported result comes back. Seven so far. Papers come out of the find sweeps by neutral criteria (nobody nominates them), and they’re deliberately ones that run on a CPU in minutes, so the result is mechanical: the number regenerates or it doesn’t.
The board
Everything lives in one repo that doubles as the database. A mission is just a
directory: mission.md for what it is, brief.md for the exact prompt a
session gets, and runs/ for whatever’s come back. A build script reads the
tree and generates the site at nightshift.benwolf.me. Nothing on it is typed in
by hand; every status and number comes from which files exist and what’s in
them.
Each open mission has a link that opens Claude Code with the brief already loaded and the right repo selected. When a session finishes, it pushes its output to an inbox repo as a pull request. A relay pulls deposits onto the board on a schedule, but only ones I’ve promoted through a workflow that names the pull request and its exact commit. Nothing a session or a depositor does on its own gets a file onto the board.
How a reproduction works
The full chain is find → review → extract ×2 → reconcile → key file → reproduce → compare → re-read → result.
- Two blind extractions. Two separate sessions read the paper and list every result in it (each coefficient, standard error and sample size) with its exact location: table, row, column. Neither one knows the other exists.
- Reconcile. A script lines the two lists up. Where they disagree on whether a number even counts as a result (a real estimate, a parameter, a restatement of something earlier), the written inclusion rules settle it. When the rules can’t, the mission stops and waits for me to rule.
- Key file. The agreed list becomes a key file of targets: locations and labels, never values. The script refuses to write one with a value in it.
- Reproduce. A third session gets the package and the key file, runs the code, and reports what each target came out as. It never gets the published number to match, and its output gets refused if it contains one.
- Compare. Matching happens on the board, outside any session, at the paper’s printed precision. Either rounding convention counts (R rounds half to even, Stata half away from zero, and papers basically never say which). One unit off in the last digit is its own class, a sign flip is always a difference, and a ×100 or ×1000 gap gets flagged as a units mismatch.
- Blind re-read. A difference isn’t a finding yet, since it could just be a typo in the extraction. Every difference goes to a fresh session that gets locations only (no values, no reason) and writes down what’s printed there. The difference only counts as confirmed if that re-read agrees with the extraction.
Blinding
The rule: a session must not be able to reach the answer. Just telling it not
to look doesn’t hold up. An early run’s own record said it hadn’t seen any
prior results, when a routine git log had literally shown it one in a commit
message, and that only came out when I asked it directly. So everything below
is built into the setup instead of written into a prompt.
There turned out to be a ton of ways in.
Container network. The extraction environment is set to “no network
access”. That stops curl and WebFetch, and it took me a week to notice it
stops nothing else. On 2026-08-16 a web search under that exact setting handed
one of REPRO-001’s headline effect sizes to a session that hadn’t even been
given the paper. The 46 blocked fetches before that had looked like a working
seal.
Web search, fetch and MCP connectors. Search and connector calls run on
Anthropic’s servers and never cross the container’s network boundary, so no
environment setting touches them. Connectors are even worse: they’re attached
by the account, not the repo, so no file in any repo can take them away, and a
cloud session loads a project’s .mcp.json without asking. One probe found
nine connector families available in a “sealed” session, including a paper
database that could have pulled the exact paper being withheld.
The hook. What actually holds is a PreToolUse hook committed in the
deposit repo’s .claude/. It refuses web search, web fetch, every tool whose
name starts with mcp__ (a prefix match, so plugin-bundled connectors get
caught too), and Bash commands that retrieve anything: commands naming an
address, package installs, and git clone/fetch/pull. It lets git add,
commit and push through, because a deposit is a push. The permission deny
rules next to it were never once seen firing, so on three of the four channels
the hook is the only thing standing there.
Checking the hook’s own claims. I tested every claim in the hook’s header
by feeding it real tool-call payloads and reading the exit code. That turned up
a real gap: the Bash rules were written on 2026-08-22, but the hook’s matcher
wasn’t sending Bash to it until 2026-09-19, so for a month Bash was covered by
the environment alone. It also caught git -C <dir> fetch and R’s
install.packages() slipping past, both closed the same day. Still open (and
written in the hook itself): a URL built inside an interpreter gets through,
since the hook reads command strings.
The environment isn’t part of the seal anymore. A dispatch link picks an
environment by name in the clicker’s own account, and a name that doesn’t
match falls through to their default, which has network. So a session could
pass its own checks while curl worked fine. Now the hook covers Bash too, and
the blinding doesn’t depend on which environment ran.
Seal check. Before an extraction starts, the session runs four probes: one web search for a phrase from the paper’s title, one fetch of example.com, one Bash request to the same address, and one connector call. All four results go verbatim into the deposit next to the session log. If any of them answers, the run stops and deposits a finding instead of an extraction. A connector just being available counts as open, whether or not anything called it.
Other readings. The two extractions get opaque tokens instead of “reader
A” and “reader B”, since telling a session it’s reader A tells it there’s a
reader B. The token-to-letter mapping never leaves the board, and the build
checks that a mission’s two briefs differ in the token and nothing else. Then
an audit on 2026-08-27 found the second reading could still read the first
one’s deposit: a git clone of the inbox pulls every branch, so it was already
sitting on disk, readable even with the network cut at the kernel
(unshare --net). Deposits now travel as pull refs, which a clone doesn’t
fetch.
The reproduce lane. A reproduction has to install packages and download data, so Bash can’t be blocked there. It runs from a separate repo whose hook still refuses search, fetch and connectors, and its README won’t call it sealed. The guarantee there is different: the session never gets a value, can’t deposit one, lists the package’s own result files without opening them, and the comparison happens somewhere it can’t see.
Still open. A paper handed over as an attachment arrives inside the message, not through a tool call, so no hook can see it; a wrong or annotated PDF would pass a clean seal check. Nothing enforces that the two readings come from different people. And a kernel-level network sandbox under the hook would back up the Bash rules and nothing else. Each of these has its own write-up with the options.
Rules and rulings
What counts as a result is decided by a versioned set of inclusion rules (F1,
F2 and so on). When two extractions disagree in a way the rules don’t cover, I
write the decision into rulings.yaml with the reasoning. A ruling that keeps
coming up across papers gets flagged as a candidate rule. Since every
extraction is stored, a new rule version can be replayed against old missions
without rerunning anything. REPRO-001 had 27 disagreements between its two
extractions, and one rule (F9) answered 21 of them on replay.
Automation
Promoting candidates from the find sweeps into reproduce missions is written as a weekly scheduled Claude routine, with its full prompt kept in the repo. It runs with every connector removed, since a routine has no permission prompts and would otherwise carry write access to everything on the account. It writes a heartbeat line on every run, quiet ones included, because a routine that’s stopped firing looks exactly like one with nothing to promote.
Where it stands
REPRO-001, a political science paper on women MPs and polarization, is the shakedown run for the whole pipeline. The first reproduction died at the first table: a September 2026 update to an R package (estimatr 2.0) removed a function the authors’ code calls. A second run pinned that one package to the last version that still had it and changed nothing else, so the pair separates “the code was always broken” from “the dependency moved”. That run came back with 32 targets equal, 2 off by one in the last digit, and 9 missing. Its extractions ran before the seal check existed, though, so it can’t count as a result. The first fully sealed result will be one of REPRO-002 through REPRO-007.
An earlier, simpler version of the pipeline ran five reproductions. The last one ran unattended on a schedule and matched 9/9 coefficients, 9/9 standard errors and 9/9 sample sizes against the authors’ own archived log.
What it’s good for
- Checking whether a published quantitative result regenerates from its own data and code, number by number, with a record of exactly what ran.
- Mapping a scientific dispute: who claims what, with what stake and what sources, and what an outsider could actually verify.
- Finding which papers are checkable at all.
- The blinding works for any agent run that must not see its answer: evals, benchmark checks, anything where a leaked result makes the output meaningless.
Stack
GitHub repos for the board, the inbox and the reproduce lane, Python for the build, relay, reconcile and compare scripts, a static site on Vercel, and Claude Code sessions for every extraction, reproduction, map and sweep.