@gram 2
doc blog-codex-memory-benchmark
t What a Real-World .gram Benchmark Measures—and What It Does Not
d A measured comparison of one reconstructed Codex project-memory file, its current-state projections, deterministic checks and publication limits.
tag research
view document
::rich
# What a Real-World .gram Benchmark Measures—and What It Does Not

A format comparison is useful only when readers can see what data and representation were measured. This benchmark examines one Codex project-memory document reconstructed from repository history. It is not a measurement of an agent remembering facts during 18 months of live work, nor an evaluation of model accuracy. The source is a final, hash-manifested report with internally self-consistent checks; it does not establish an independent external reproduction.

## Corpus and method

The analyzed .gram container is 1,046,361 bytes and contains 18 states, revisions 0 through 17, and 532 records at its final state. The current .gram source projection is 219,837 bytes; the remaining 826,524 bytes are History/6 journal overhead. Its record set includes project, goal, source, history, requirement, assumption, event, evidence, component, decision, task, risk, metric, blocker and artifact types. The reconstruction spans source history from 2025-04-16 through 2026-10-10. It is retrospective repository-history reconstruction, not organically maintained personal memory.

For current-state size comparison, the current .gram source is compared with a chosen reversible Markdown schema and JSON representations of the same normalized persisted semantic state. Current Markdown is 309,305 bytes; minified JSON is 291,140 bytes; pretty JSON is 466,852 bytes. On these selected encodings, the 219,837-byte current .gram source is 28.9% smaller than Markdown and 24.5% smaller than minified JSON. These are conditional results for this corpus and schema, not intrinsic Markdown or JSON overhead and not a universal storage win.

The schema preserves field values, links, order and the raw rich body; semantic type codes are expanded while original field keys remain. It does not preserve source-byte identity. Do not compare the 1,046,361-byte full .gram container with a Markdown or JSON current snapshot as if they had the same scope: the full file includes prior-state journal data.

Separately, exported historical Markdown and JSON totals include per-revision effective/recorded/actor/request metadata and sum to 2,742,707, 4,161,950 and 2,569,734 bytes for Markdown, pretty JSON and minified JSON. Those totals describe 18 state exports under this scheme, not proof that snapshot formats cannot store history.

## What was checked

The private benchmark run manifest hashes 108 generated artifacts; it is not a manifest of 108 corpus inputs. Verify-only mode re-checks those artifact hashes, the raw rich body of each revision, the 532 current records and Markdown/JSON semantic round-trips for all 18 revisions. The published report states the round-trip result, but the verification output itself is not published.

Seven deterministic content oracles query selected facts such as project-head counts, an implementation-language decision, architecture links and a latest analyzed change. Historical boundary checks verify that revision 0 contains 12 records, that a later decision appears at revision 2, and that an earlier decision changes status across revisions. These are checks of persisted data, not LLM tests, answer-quality studies or accuracy scores.

The benchmark cites a pinned format revision and a native runtime fingerprint. Independent reproduction would require the corresponding source and runtime evidence; a reader without those exact inputs cannot assume they will produce the same report. The benchmark did not independently fetch an upstream repository checkout. Public repository citations provide provenance for project context, not proof that this reconstructed sample represents all Codex users.

## Interpretation and practical value

For product and architecture readers, the example shows how one structured file can combine current prose, linked records and inspectable prior states. A tool can ask what changed, when a decision appeared or what evidence supports a task without relying on separately maintained snapshots. A compact source may also be convenient to store or move. This benchmark does not prove lower model token usage, better retrieval, reduced inference cost, higher factual accuracy or performance over a carefully designed event store.

The useful lesson is methodological: label current and historical scopes separately, disclose encoding choices, retain provenance and test exact historical boundaries. Before applying this size result to another project, measure its content, structure, rich-body size, chosen schema and history retention. Record density, repeated fields, prose, link patterns and export conventions can change the outcome substantially.

## Evidence access and further reading

This site includes privacy-redacted report and oracle exports at /evidence/codex/report.json and /evidence/codex/oracle.json. They are publication exports, not byte-identical copies of the original files. The manifest at /evidence/codex/manifest.json lists the two published exports and distinguishes publication hashes from original-source hashes; it is not the private 108-artifact run manifest or a full reproduction package.

The original corpus, source snapshots and verify-only output are not part of these exports. These materials let readers inspect selected report and expectation data; they do not show that this article independently reran the benchmark or that an external reader reproduced it. The internal benchmark source and format specification were reviewed for methodology and semantics but are not publicly available.

[[src-benchmark-run]]
[[src-benchmark-review]]
[[src-format-history]]
[[src-codex-source]]
[[evidence-benchmark-oracles]]
[[knowledge-benchmark-scope]]
[[knowledge-benchmark-generalization]]
::end
SRC src-benchmark-run
t Final Codex memory benchmark report
d Report and oracle export provenance reviewed 2026-10-12; public copy is privacy-redacted and differs from original bytes.
ref /evidence/codex/report.json
SRC src-benchmark-review
t Benchmark research source package
d Internal untracked benchmark research material inspected 2026-10-12; not publicly available and not cited at a commit that lacks it.
SRC src-format-history
t Thegram persistent format contract
d Internal format source at revision 625edec; not publicly available.
SRC src-codex-source
t OpenAI Codex public repository
d Public repository cited for provenance; benchmark did not independently fetch its checkout.
ref https://github.com/openai/codex
E evidence-benchmark-oracles
t Seven deterministic content oracles
d Seven persisted-data queries and historical boundaries; they do not measure LLM accuracy.
source src-benchmark-run
K knowledge-benchmark-scope
t Compare equal current-state scopes
d Size comparisons use 219,837 current .gram bytes and selected reversible Markdown/JSON current projections, not full journal against a snapshot.
source src-benchmark-run
K knowledge-benchmark-generalization
t Avoid universal format claims
d One retrospectively reconstructed Codex corpus and selected schemas do not establish universal size, retrieval or model-accuracy performance.
source src-benchmark-review