Research, results & downloads
What the current evidence establishes, where it falls short, and how to reproduce it.
Current status: disclosed development studies. The results below come from ordinary deterministic programs. No independent participant or external AI-model result has been established. The purpose is to make the diagnostic inspectable, expose its weaknesses, and enable a useful independent test.
The question we want to establish
When an inherited procedure keeps passing its usual checks, can an agent detect that it has become unreliable—and choose an economical experiment to decide what to do?
This matters because a familiar check can miss the property needed for the next prediction. A useful diagnostic should reveal that failure and help a developer choose a better agent. We still need independent evidence that this actually changes such a decision.
1. Information transfer: what the hosted game tests
The browser game uses a known finite cubic process, an affine instrument and exact calibration. An earlier model or its equivalent raw measurements can inform target experiments. The matched follow-up used 32 disclosed worlds, three information arms and five budget caps: 480 records, with correlated outputs and repeated comparisons within worlds.
| Information | Brier loss | Target measurements | Source + target |
|---|---|---|---|
| Source model | 0 | 1.53125 | 7.53125 |
| Equivalent raw source data | 0 | 1.53125 | 7.53125 |
| No source information | 0 | 3.0625 | 3.0625 |
Source information saves target measurements for this algorithm. Including six source measurements makes one source-informed recipient more expensive overall. Model and raw-data arms have identical actions, predictions, stopping and costs in every paired record. No artifact-format advantage or same-source repeated-reuse saving was demonstrated.
The follow-up was designed after inspecting earlier results and reused public fixtures. Both arms reach zero loss in a small, noiseless known family. The experiment-selection rule is greedy observation entropy, not proven optimal or directly optimized for forecast loss.
2. Procedure validity: a separate local study
This dynamic procedure diagnostic is included in the download. It is separate from the hosted cubic game. Its inherited check measures an almost settled response; a changed two-stage mechanism passes while early responses differ.
| Command update | Inherited prediction | Actual response |
|---|---|---|
| 1 | 0.5 | 0 |
| 2 | 0.75 | 0.25 |
| 3 | 0.875 | 0.5 |
| 4 | 0.9375 | 0.6875 |
After 32 updates, the normalized reading is about 0.999999992, inside the inherited check's acceptable range [0.95, 1.05]. That check misses the early-response error.
| Ordinary program | Invalid accepted / 180 | Valid rejected / 12 | Mean physical MSE | Target units |
|---|---|---|---|---|
| Blind reuse | 180 | 0 | 0.368564614 | 0 |
| Always reject; retain source prediction | 0 | 12 | 0.368564614 | 0 |
| Inherited settled-response check | 84 | 0 | 0.356866368 | 40 |
| Check, then validate and repair | 0 | 0 | 0 | 54 |
| Fixed system identification | 0 | 0 | 0 | 16 |
The short fixed identification script solves every full-access case. Source production adds 16 units per run, so validation/repair totals 70 and fixed identification totals 32. This family provides a regression diagnostic; it does not establish frontier difficulty or a resource advantage for transferring a procedure.
Two additional restricted cases cannot be distinguished by any legal experiment, although later outcomes differ. Their warranted decision is undetermined. A lucky hidden-label guess does not resolve the information limit. All 194 configurations across five programs produce 970 records; the 24 outputs per run are not independent trials.
What a researcher should challenge
- Ability measured: forecast scores and internal consistency do not establish causal understanding, explanation quality or discovery of a new method.
- Generality: finite, exact, disclosed families can reward specialized solvers. New seeds are not new mechanism families.
- Transfer claim: the donor artifact was produced by the referee; equivalent raw data carries the same information. Independent method invention is untested.
- Cost claim: report source acquisition and target experiments together. Compute, tokens, money and human design effort were not measured in these studies.
- Validation: independently implemented internal verifiers are useful checks, but external replication and independent-user usefulness remain missing.
- Selection: the worked run is chosen for explanation. The complete disclosed archives, including unfavorable comparisons, are in the download.
Related work and novelty limits
DiscoveryWorld already evaluates experimental investigation and scientific reasoning. DiscoverPhysics evaluates recovering hidden physical laws through experiments. ExpGraph's first-party description overlaps with budgeted interventions, mechanism repair and paired campaigns; its implementation and results have not been independently verified here.
A simulated science game or paired reporting workflow alone does not establish novelty. These are close comparisons, not an exhaustive priority search. A specific useful distinction still needs evidence.
Download and reproduce
The curated ZIP contains the scientific source, all three disclosed study archives, verification scripts, worked-run reports and the prospective independent-pilot protocol. It is available without GitHub access. The public research repository provides the reviewed source snapshot, documentation and disclosed records. This downloadable ZIP is the earlier scientific reproduction bundle; its manifest identifies its contents.
Extract, open the unknown-engine-research folder, and use Python 3.10 or later. No install or model key is needed:
python scripts/verify_public_example.py python scripts/verify_transfer_preview.py evidence/transfer-preview python scripts/verify_transfer_matched.py evidence/transfer-matched python scripts/verify_procedure_gate.py evidence/procedure-gate
The README includes regeneration commands and a runnable local comparison. All code and included research records are Apache-2.0; license notices accompany the export. Hashes verify consistency, not identity, a trusted timestamp or referee honesty.
The next independent test
- An independent developer names a real decision and their existing alternative test.
- Freeze the current agent and a version with explicit procedure checks, the case allocation, success criteria, budgets and all retry rules.
- Retain every attempt; report invalid acceptance, valid rejection, uncertainty, execution failures, prediction error and full costs separately.
- Confirm the change on a separately authored new mechanism family against strong ordinary baselines.
- Record whether the report changed the developer's decision, including null or negative results.
Status: prepared, not run. The supplied local campaign tool supports the existing family. A new-family adapter, independent participants and independently operated confirmation are still needed. The downloadable protocol includes a freeze worksheet and analysis requirements.
If you have an agent and a concrete decision this could help with, describe your current test and what you would want to compare in the moderated comments. Reproduction reports, simpler solutions and objections are welcome. Keep credentials, private prompts and withheld answers out of comments.