# t03 vs IB on the real engine (partial: 144 of 1,024 seeds)

**Headline.** On seeds 9500000..9500143 (n=144, not the 1,024 requested), t03 minus IB is +0.7448 edge per seed (se 0.0836, paired). t03 mean edge is 534.60 and IB 533.86. The 1,024-seed run did not fit the time limit (see Problems).

## Setup
1. Cloned `https://github.com/benedictbrady/amm-challenge` and checked out `6b6921f8d463314aa7c96234e1be25a25245f704` (HEAD "Merge pull request #6 ..."). Location: `test/scratch/amm-challenge` inside the workspace, not /tmp.
2. Saved the given script verbatim as `scratch/setup.sh` and ran `sh scratch/setup.sh` once from the checkout root. Every step succeeded:
   - wheel sha256 check passed (`aedb6828...`)
   - `uv venv --python 3.12`
   - installed amm-sim-rs 0.1.0.post1, numpy 2.5.4, numba 0.68.1, cma 4.5.0, py-solc-x 2.1.0, pyrevm 0.3.7
   - editable install of amm-challenge
   - solc 0.8.24
   - `amm-match validate contracts/src/StarterStrategy.sol` printed "Strategy 'StarterStrategy' validated successfully!"
3. Later commands were prefixed with `. scratch/env.sh &&`.

## Artifacts
| artifact | url | sha256 of file | sha256 of decoded `code_zb64` source | bytes |
|---|---|---|---|---|
| t03 | https://api.imd.fun/artifacts/2333ecf28d4356bd232b85c1e55842e94e1ada05e1cd067830b5625dfdbc5336 | 2333ecf28d4356bd232b85c1e55842e94e1ada05e1cd067830b5625dfdbc5336 | cb6d945511e1e10af78baa64c2a6cb25806f0998403ce6440654bfc3c70092d9 | 27712 |
| IB | https://api.imd.fun/artifacts/afd5183e9ff3235334e041c00b0cd19079599a2c6464db95f8c1586712105557 | afd5183e9ff3235334e041c00b0cd19079599a2c6464db95f8c1586712105557 | 301941829a44bb60977bb28181a5e4bfa6544f3696763f987054789b2611d686 | 27524 |

- Decoding: base64, then zlib. For t03 the brief's "sha256 cb6d9455..." is the decoded source hash, not the file hash. The file hash equals the hash in the URL.
- `amm-match validate`:
  - t03: "Validating strategy... Compiling strategy... Strategy 'censored_10' validated successfully!"
  - IB: "Strategy 'inputs_b' validated successfully!"

## Results
Each strategy runs alone against the 30 bps VanillaStrategy through `amm_sim_rs.run_batch(X, VanillaStrategy, configs, 8)`. Configs are built as in `MatchRunner._build_configs` (config i uses `default_rng(i)` and sim seed i, with i = the seed). The score is the strategy's "submission" edge.

| strategy | n seeds | mean edge | sd | se |
|---|---|---|---|---|
| t03 (censored_10) | 144 | 534.6038 | 92.93 | 7.745 |
| IB (inputs_b) | 144 | 533.8589 | 93.00 | 7.750 |

Seeds: 9500000..9500143 (6 segments of 24, each strategy run alone per segment). The segments are in `scratch/state/*.jsonl`, which is not delivered.

## Paired (t03 minus IB, same seeds)
- n = 144
- mean_diff = +0.7448
- sd = 1.0027
- se = 0.0836
- worst5 (most negative for t03):
  - 9500017: -2.6096
  - 9500088: -1.9770
  - 9500117: -1.7480
  - 9500043: -1.2055
  - 9500127: -1.0736
- Best5 for t03:
  - 9500003: +5.1746
  - 9500004: +4.7400
  - 9500007: +3.0022
  - 9500079: +2.9659
  - 9500101: +2.7729

## Gas
Submission role only, from `amm_sim_rs.gas_stats()`. `gas_stats_reset()` ran before each segment, so counts are summed over segments and the max is taken over segments. OutOfGas, revert, other halt, tx_error and bad_return were 0 for both.

| strategy | afterSwap calls | afterSwap peak | afterSwap mean | OOG | afterInitialize max |
|---|---|---|---|---|---|
| t03 | 1,572,040 | 235,394 | 135,477 | 0 | 243,234 |
| IB | 1,559,056 | 234,805 | 135,147 | 0 | 243,227 |

Both afterSwap peaks are under the 245,000 limit. Baseline (Vanilla) afterSwap max was 22,592 in the early 8-seed probe.

## Timing
- About 12.0 s per seed for t03 and 11.6 s per seed for IB, measured on the `run_batch` call only (compile and validate excluded).
- A 24-seed segment took about 5.7 min wall including compile.
- This machine has a cgroup `cpu.max` of 100000/100000, i.e. 1 CPU. The 8 workers therefore gave no real parallelism: user time was about equal to wall time.

## Harness (verbatim: scratch/engine_harness.py)
```python
#!/usr/bin/env python
"""Engine harness: python scratch/engine_harness.py <strategy.sol> <first_seed> <count>
Runs the strategy alone vs the 30 bps VanillaStrategy via amm_sim_rs.run_batch(..., 8),
configs per MatchRunner._build_configs (config i uses rng seed i and sim seed i, here i=seed).
Prints one JSON line per seed {"seed","edge","edge_norm","pnl","pnl_norm"} then a final
{"summary":..., "gas_stats":..., "seconds":..., "sec_per_seed":...} line."""
import sys, json, time
import numpy as np
import amm_sim_rs
from amm_competition.competition.config import BASELINE_SETTINGS as B, BASELINE_VARIANCE as V
from amm_competition.evm.baseline import load_vanilla_strategy
from amm_competition.evm.compiler import SolidityCompiler
from amm_competition.evm.validator import SolidityValidator


def build_configs(first, count):
    out = []
    for i in range(first, first + count):
        rng = np.random.default_rng(seed=i)
        size = rng.uniform(V.retail_mean_size_min, V.retail_mean_size_max)
        rate = rng.uniform(V.retail_arrival_rate_min, V.retail_arrival_rate_max)
        sig = rng.uniform(V.gbm_sigma_min, V.gbm_sigma_max)
        out.append(amm_sim_rs.SimulationConfig(
            n_steps=B.n_steps, initial_price=B.initial_price, initial_x=B.initial_x,
            initial_y=B.initial_y, gbm_mu=B.gbm_mu, gbm_sigma=sig, gbm_dt=B.gbm_dt,
            retail_arrival_rate=rate, retail_mean_size=size,
            retail_size_sigma=B.retail_size_sigma, retail_buy_prob=B.retail_buy_prob, seed=i))
    return out


def main():
    path, first, count = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
    src = open(path).read()
    v = SolidityValidator().validate(src)
    assert v.valid, v.errors
    c = SolidityCompiler().compile(src)
    assert c.success, c.errors
    van = load_vanilla_strategy()
    cfgs = build_configs(first, count)
    amm_sim_rs.gas_stats_reset()
    t = time.time()
    res = amm_sim_rs.run_batch(list(c.bytecode), list(van._bytecode), cfgs, 8)
    dt = time.time() - t
    for i, r in zip(range(first, first + count), res.results):
        print(json.dumps({"seed": i, "edge": float(r.edges["submission"]),
                          "edge_norm": float(r.edges["normalizer"]),
                          "pnl": float(r.pnl["submission"]), "pnl_norm": float(r.pnl["normalizer"])}))
    print(json.dumps({"summary": {"path": path, "first": first, "count": count},
                      "gas_stats": amm_sim_rs.gas_stats(), "seconds": dt, "sec_per_seed": dt / count}))


if __name__ == "__main__":
    main()
```
Use: `. scratch/env.sh && python scratch/engine_harness.py <strategy.sol> <first_seed> <count>`.

## Problems
- **Seeds not completed.** The brief asked for 1,024 seeds per strategy. At about 12 s per seed on 1 CPU that is about 3.4 h per strategy, against about 114 min for the whole task. I ran 144 paired seeds (9500000..9500143) instead, which is 14% of the request. Seeds 9500144..9501023 were not run. Later items should not treat them as used.
- **Sampling error.** The se of 0.084 comes from n=144 only. The paired t03-IB gap (+0.74, about 8.9 se) is clearly nonzero on these seeds. The worst5 are the worst of 144, not of 1,024.
- **8-seed probe.** An early 8-seed probe of t03 on seeds 9500000..9500007 ran before the segments. It was a harness test, and those seeds are re-run inside the first segment.
- **Hash note.** The brief's t03 "sha256 cb6d9455..." matches the decoded source, not the artifact file.