AI resistance lab devlog: from visual complexity to measured effort

The first bug was treating a proxy as a result

Output size, changed locations, virtual-machine domains, fragments, bridges, and bytecode counts are real observations. They describe transformer output, not recovery. A high similarity score can mislead in either direction: a layout may change dramatically while semantics remain easy to infer, or a small change may break one analysis assumption.

The threat model therefore assumes a capable analyst with the complete artifact, a disassembler, a decompiler, a debugger, traces, symbolic execution, batch samples, an automated devirtualizer, and an AI agent. This is intentionally stronger than a wrapper recognition test. It makes the target measurable: reduce the stability and transferability of convenient representations, then count the semantic work an external system actually completes. Recovery remains theoretically possible because the processor must eventually execute meaning.

A related admission failure made the same lesson concrete. Several research families were labelled hardening-rejected while ordinary production entry points could still emit them. The fix added hard-fail gates to ordinary masked native-island, coded-multiplication, exact Boolean, and ZeroPredicate entry points. The automatic word candidate remains compiler-private, with no production selector. The production path now rejects those candidates before publishing a PE or sidecar; old research candidates remain available for controlled functional, unwind, and counterexample research. Current admission regressions preserve that boundary and do not replay a historical attack as a fresh resistance measurement.

There are bounded counterexamples outside the six-variant scoreboard. An independent A-only agent recovered an initial formula from a frozen masked artifact in 16 minutes 50 seconds, exceeding its requested 10-minute budget; the frozen predictor later matched 2,048 new opaque-B inputs plus edge cases. A separate single-trace word-candidate extractor built a 34,242-node model from A and matched B on 2,048 checks. These are observed transfer results with explicit scope, not universal proofs or completed campaign measurements. They are why the production gates stay conservative. For the deeper semantic boundary, see the Aegis semantic-boundaries devlog.

Freezing the six-variant experiment

The first implementation track is the existing aegis lab flow. It starts from one bounded ground-truth case and produces six named artifacts: unprotected, basic_obfuscation, single_vm, diversified_vm, dscf, and multi_vm_dscf. Naming the variants is a small but important data-model decision. A later observation can identify exactly which transformation it evaluated instead of relying on an informal description of a binary.

CTest executes the six artifacts in independent processes and checks original-versus-protected behaviour. The reverse challenge adds two Maximum-seed outputs that must preserve behaviour while differing as artifacts. The repository’s previous artifact-only extractor must fail at wrapper recognition, and its single-field decoder must not recover the protected Critical or Internal text. That is useful evidence that a recorded extractor’s assumptions do not transfer for free. It is not evidence that an unknown or adaptive AI agent cannot build a new extractor.

Ground-truth JSON is bounded at 1 MiB, 4,096 cases, four parameters per case, and constrained depth, turning malformed input into a parsing result.

External analysis is a separate measurement surface

Once the artifacts were reproducible, the missing component was an actor that actually performs reverse engineering. aegis lab-observe accepts a complete observation from an agent environment or analyst. The required fields cover function, CFG, and state accuracy; pseudocode similarity; cross-build transfer success; tool calls; elapsed milliseconds; token use; human intervention; and semantic success.

The context is part of the observation, not an annotation added later. Input, seed, tool version, agent or model version, prompt and tool policy, time and token limits, and analysis environment must be fixed. Without those controls, a lower token count can simply represent an easier prompt, a stronger tool, or a different definition of success.

observation = {
  variant: <one of the six frozen names>,
  context: { tool_version, agent_version, prompt_policy },
  accuracy: { function, cfg, state, pseudocode },
  cost: { elapsed_ms, tokens, tool_calls },
  human_intervention,
  semantic_success
}

The schema is illustrative, not a private implementation dump. Within the six-variant lab and unfinished COST_LOCAL campaign, its most important field is the absence state: until an observation is imported, external_ai_measurements_present stays false and the variant’s ai_observation stays null. That scoped null does not erase separate A-only counterexamples; it keeps them from being mislabelled as a complete six-variant benchmark.

The summarizer refuses to make a missing run look complete

aegis lab-summarize strictly parses the bounded observation database, selects a run by ID, and aggregates only variants with actual observations. It can report means and success rates, plus unit-labelled measures such as semantic recovery per 1,000 tokens and semantic recovery per minute.

It does not interpolate a missing variant, translate a seed into an AI score, or use static complexity as a substitute for recovered meaning. Semantic Recovery Efficiency must use recovered ground-truth semantics and a disclosed cost denominator. A comparative result also needs confidence intervals and failure cases. The table should show what was measured, what was not, and which comparison cannot yet be made.

COST_LOCAL protects the record around the experiment

The six-variant lab defines what an external evaluation should observe. COST_LOCAL asks whether a larger, adversarial campaign can preserve those observations without allowing a plausible local file set to promote itself. Its current status is implementation in progress and not promotion-eligible. Local tests establish serialization, authentication, replay, and fail-closed parsing properties; they do not substitute for a deployed service audit or a real 10-pair/24-hour adversarial campaign.

The harness separates public A material from generator-private B material, gives coordinator roles independent roots and HMAC domains, and keeps numeric projection deliberately non-authoritative. A canonical receipt binds campaign and freeze context, run and attempt, pair, scoreboard and window, artifacts, candidate and dependency data, monotonic interval, verdict, and resources. The chain then checks contiguous sequence, predecessor recomputation, terminal binding, and duplicate rejection.

This is a custody problem as much as a schema problem. Authenticated Accepted and InvalidRun events propagate through conservative latches. A write-through journal, record digest, and replay path detect incomplete records, while rollback resistance is explicitly false. The harness can establish that a local record is internally consistent; it cannot establish that an older, locally valid file set was never restored.

The inventory makes the gap visible. Schema3 precommits 480 pair, subject, stage, scoreboard, window, and coverage slots. Authenticated finalization binds the full inventory and exact chain tip, seals after flush, and verifies the end of the journal on replay. A complete local chain is still different from a complete campaign because every context-bound result and the external authority around it must exist.

The executable corpus became a semantic reference, not a resistance score

In parallel, the placeholder corpus was replaced with fourteen separate executable shapes covering control flow, data dependence, arithmetic, parsing, state, buffers, and ABI or exception behaviour. The release validates 168 independent oracle comparisons. PE discovery proves extents from chained unwind fragments and closed metadata-less leaves; direct-export and resolved-semantic instruction views classify with Other=0.

The machine-effect executor now covers registers and flags, bounded memory and heap, vector state, interprocedural calls and returns, exact import contracts, and bounded MSVC exception handling. Exact results can be stored as fourteen-entry manifests and summarized into seven categories. This is the reference layer needed to tell whether a future protected artifact preserved known behaviour.

It is not yet a protected production path. The accumulator-only route still rejects these shapes for bytecode or PE emission, and no hidden service holdout has earned a resistance result. The corpus can answer whether an implementation agrees with an oracle; it cannot answer whether an agent recovered the oracle, how much effort that required, or whether the recovery transfers to a fresh build.

Failure modes remain first-class outputs

The implementation records failure instead of smoothing it into a success-shaped report. A missing external observation remains null. A sequence gap invalidates the admitted receipt chain. Duplicate, forked, tampered, or truncated campaign records are rejected. Generation outcomes distinguish failed, succeeded, and not-run states, require reasons or diagnostics, and reject silent resampling.

The same discipline applies to the campaign’s current scope. The local result, receipt, view, and scoreboard chain includes synthetic production-path coverage and an unprotected live reference run. Protected A/B execution, service-private fresh holdouts, and per-category reporting remain open. A finalization flag cannot promote those placeholders into the 480-result matrix.

The next run must measure recovery, then survive replay

The next meaningful checkpoint is not another complexity count. It is a complete, context-bound set of external observations and the custody evidence around them. Restricted A, B, baseline, coordinator, signer, vault, and executor roles need separate service identities. Generation schedule and outcome issuance need to move behind the restricted builder path, and the persistent roots and campaign ledger need an external monotonic rollback anchor.

Then the executable corpus must run through protected A and opaque B for all ten pairs, with fresh service-private holdouts and per-category reporting. Job Object measurements need to enter receipt v3, including cumulative 30-minute, 4-hour, and 24-hour state, crash and clock-gap invalidation, and the primary effort metric. The deployment audit still needs service image hashes, restricted runtime tokens, pipe identity, handle census, denial tests, and normalized error and timing channels.

Only after those records exist can a second runner replay the frozen corpus, tool and agent policy. A repeated comparison can then report semantic recovery, transfer, cost, uncertainty, and failure cases. Until that happens, the strongest accurate result is narrower: Aegis has a reproducible six-variant lab vocabulary, explicit admission failures for known broken production families, bounded counterexamples that inform the threat model, a strict external observation boundary, a 14-shape semantic reference, and an evidence ledger that still says the campaign is unfinished.