Hermes Labs

Program or investigation overview

Artifact overview

Part of the inspectable closed-world token apparatus. Engineering checks and source matching limitations are distinct from empirical model findings.

Research status
Offline evaluation apparatus
Updated
Source version
0.1.1 | Created: 2026-10-11 | Updated: 2026-10-12 | Status: proposed evaluation; offline reference implementation
In this record

Scientific record

Project: Project Fiction, Hermes Labs

Epistemic labels: CURRENT HYPOTHESIS; NEW PROPOSAL; UNRESOLVED

Fiction-CWR measures goal-directed closed-world recombination: produce an executable solution from supplied primitives, exact semantics, constraints and budgets when an analogous composition motif has or has not been demonstrated. It tests existing AI capability related to FH-01/FH-07; it does not establish an innovation event, human-relative deficit, absence from pretraining, or reward suitability.

Recorded model runs: zero. EXP-0002 is a proposal. Eight concrete showcase specifications are a directional starting set, not eight implemented generator families or independent confirmation. The token workshop is implemented, with a bounded whole-solution audit and one imported public F/C candidate. The other seven full checkers and private structural generators remain proposed. The candidate retains measured source mismatches and is not frozen for model execution. Verification records actual offline checks after execution.

Eight deliverables

DeliverableVersioned artifact
Falsifiable hypothesis and capabilityDesign, canonical hypotheses
Retrieval, execution, familiar and withheld synthesis contrastsDesign, task catalog, prompt specification (not included in this public selection: PROMPTS.md)
Objective exact-validity scoringGrader, reference checker, independent oracle
Eight concrete candidate tasksTask catalog, token fixture, private witness (not included in this public selection: fixtures/private/token-workshop-001.witness.json)
Frontier model reproducibility, statistical and budget planModel plan (not included in this public selection: MODEL-PLAN.md), proposed configuration (not included in this public selection: config/run-plan.json), run record template (not included in this public selection: config/run-record-template.json)
Failures, alternatives and analysisAnalysis (not included in this public selection: ANALYSIS.md), prior art (not included in this public selection: PRIOR-ART.md)
Hackathon submission and implementation stepsSubmission draft (not included in this public selection: SUBMISSION.md), roadmap (not included in this public selection: ROADMAP.md)
Hermes integration and durable recordsIntegration (not included in this public selection: INTEGRATION.md), experiment record

Run the offline reference

Requires Python 3.10+ and the standard library only. No installation or network is needed. Run from this directory:

python -B -m unittest discover -s tests -v
python -B verify_showcases.py
python -B harness.py inspect fixtures/token-workshop-001.json
python -B harness.py solve fixtures/token-workshop-001.json
python -B harness.py grade fixtures/token-workshop-001.json fixtures/private/token-workshop-001.witness.json

The witness and oracle are evaluator-only. Model-facing prompts receive only the declared world and selected demonstration materials. Inspect the CLI usage if a command is changed during implementation. Expected witness grade: PASS; the independent solver finds a valid four-action solution. Optional explanations are absent from the strict scored object and remain unscored if recorded separately.

What must happen before empirical use

Build and validate matched F/C stimuli with structural holdouts and independent fixtures; implement the remaining selected checkers; freeze task hashes, inference scope, numerical criteria, settings and analysis; verify official provider models/prices at execution time; obtain the user's budget authorization. Roadmap (not included in this public selection: ROADMAP.md) supplies the concrete acceptance conditions. No provider adapter, API calls, classifier training, RL or external upload is implemented in this phase.

All source claims are scoped in prior art (not included in this public selection: PRIOR-ART.md); current official hackathon guidance (not included in this public selection: SUBMISSION.md) was checked on 2026-10-11. The schedule conflict is retained. Pre-event Hermes research and this 2026-10-11 addition must be disclosed separately from work performed during the event.

Event timing update — 2026-10-13: The user confirms the October 30 kickoff and the need to build during the event. Live Build Plan (not included in this public selection: LIVE-BUILD-PLAN.md) records preparation and event work; submission guidance (not included in this public selection: SUBMISSION.md) was rechecked on October 13. Current code remains pre-event work, with no event-period additions or model runs. Earlier-work disclosure is required; exact kickoff hours and the closing deadline still differ between official pages.

Token composition audit v0.1.1

Matching and scientific scope · auditor (not included in this public selection: audit_token_pair.py) · tests (not included in this public selection: tests/test_token_pair.py) · pair manifest (not included in this public selection: config/token-pair-manifest-v0.1.1.json) · F model material (not included in this public selection: TOKEN-PAIR-F-v0.1.1.txt) · C model material (not included in this public selection: TOKEN-PAIR-C-v0.1.1.txt).

The next offline gate now certifies all 121 TW-001 sequences and both valid solutions, and reuses one EXP-0003 public development pair without importing its workflow intervention. Each target's 87,381-sequence grammar is covered by independently checked invalid-prefix pruning: 77 full-state comparisons and two accepted eight-action solutions. Source F/C worlds have 2/13 valid plans, so source matching is incomplete. The dependency is supplied in rules/primitive evidence and withheld only from C's multistep trace.

Run only the new checks from this directory:

python -B -m unittest discover -s tests -p test_token_pair.py -v
python -B audit_token_pair.py

The auditor prints evaluator-only plans/certificates. Send evaluated models only the exact F or C text listed in the manifest, never the full package or auditor output. No provider adapter or model calls exist. Actual results (not included in this public selection: verification/token-pair-audit-v0.1.1.json) and 12-test transcript (not included in this public selection: verification/token-pair-tests-v0.1.1.json) retain scope/provenance. Prior record capsule (not included in this public selection: verification/prior-records-v0.1.0.json) preserves exact pre-integration bytes; older snapshots and verification records remain intact.

Source History & Provenance

Publication and source

Public record
Selected public export · Scientific maturity is recorded separately.
Repository source
projects/fiction/evaluations/fiction-cwr-v0.1/README.md
Website snapshot
2026-10-25 (UTC)
Source date
2026-10-12 (Updated)
Version
0.1.1 | Created: 2026-10-11 | Updated: 2026-10-12 | Status: proposed evaluation; offline reference implementation

Download original file

The downloadable file preserves the selected source bytes.

Byte integrity

Download SHA-256: 27496832d8108ba129102c33abb37b538d7726c2329941bab172fc4d2ad76a27

The checksum covers the exact downloadable bytes, not the rendered HTML. It supports byte comparison; it is not a signature or an independent validation of the research.

Historical statements retain their original scope and dates. A source discussion or internal decision does not establish an empirical result. Unselected or restricted records remain unavailable in this export.