Weight-Space Programming Explained

Weight-Space Programming is the name for a general-purpose large language model reading and writing a neural network's weights directly. Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab of MIT CSAIL reported it on November 2, 2026, in the blog post "Sparks of Weight-Space Programming." OpenAI's GPT-6 Astra edited weights from natural-language instructions alone, with no gradient descent and no optimizer. The headline result: rewriting just 0.016% of Qwen3.5-0.8B's weights (131,072 values) raised its MATH-500 accuracy from 7.9% to 48.1%. Most of that gain comes from removing endless looping, not from new math skill, and the authors report that the same approach did not help a different small model, Gemma 3 270M.

This article summarizes the primary sources (the blog post, the GitHub repository of experiment logs, and the Hugging Face model card): what was shown, how it works, how to try the edited model, how it differs from existing methods, and where to be careful. Numbers come from the blog and the model card unless noted, and we did not verify them independently.

The blog defines it as a frontier LLM reading and writing a network's weights without an optimizer, and sometimes without any feedback from the environment. Normally you change a model's behavior by preparing data, choosing a loss, and updating weights step by step with gradient descent. Here the LLM (Astra) reads the numbers themselves, analyzes them with Python, and outputs rewritten weights. Weights are treated not as a by-product of training but as a program the LLM reads and writes.

The post has eight experiments in four kinds: reading a network to identify it (1 to 3), repairing a broken policy (4), writing networks from scratch (5 and 6), and editing language models (7 and 8).

In Experiment 4, the original PPO network scored 0% in 256 evaluation episodes, and a single edit with no tools produced 1,024 successes in 1,024 episodes. In Experiment 5, Astra got only the rules, source code, and network layout, with no data, loss, gradients, tools, or rollouts. Outside Breakout it did not beat long-trained PPO: Seaquest 53.1 (PPO at 500M steps: 88.9) and Asterix 53.2 (55.2).

Experiment 8 got the most attention. Astra was given 7,469 GSM8K and 7,275 MATH training problems as practice (no test sets) and a budget of up to 50 forward passes in a Python sandbox. The base model hit the 32,768-token output limit on 85 of its first 96 practice answers, so its main failure was looping without ever finishing an answer.

Astra traced the looping to a single direction written by the MLPs of layers 15 and 19, then subtracted that direction from 64 output columns of down_proj in each layer. Only two tensors differ from the base (model.language_model.layers.15.mlp.down_proj.weight and layers.19.mlp.down_proj.weight): 131,072 weights, about 0.016%. Evaluation used temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, and a 32,768-token limit, with four samples per problem on MATH-500. The post says six other benchmarks were evaluated but reports only three in the table.

A negative result is reported just as prominently. On Gemma 3 270M, MATH-500 stayed between 7.0% and 7.7% after editing, against a 6.8% base. Gemma rarely loops (8.5% hit the limit), Astra found ordinary reasoning mistakes, and no single edit moved the score by more than about a point. The authors conclude that the editing could not add an ability the model lacked.

According to the logs repository, the model was OpenAI's gpt-6-astra run through Codex CLI with built-in tools (shell, file editing, web search, sub-agents) disabled. In experiments 3, 6, 7, and 8, the model writes one Python block per turn, which runs in a sandbox with no network access and a time limit. An audit of the event streams recorded all 1,125 turns as tool-free.

- Allowed: reading weights, numpy and PyTorch computation on CPU, forward passes within a budget, and edits interleaved between passes
- Not allowed: training, gradients, fitting to data, search, rollouts, and running the network outside the budget
- Example budgets: 0 to 50 passes for Go (Exp. 6), 10, 25, or 50 for Qwen (Exp. 8), up to 100 runs for Exp. 3, up to 20 runs and 16 texts for Exp. 7
- Per-experiment limits: Exp. 4 was a single reply; Exp. 5 was fully tool-free, one-shot

The forward passes are used to evaluate, not to minimize a loss. Astra looks at the outputs, forms a hypothesis, and rewrites weights. In the Go experiment with 0 forward passes, a single 336-line Python run produced a network that won more than half of 2,000 games against each of two PPO networks (the exact win rate is not given). With 50 passes it beat a PPO network trained on 20 million games. However, the opponents are smaller networks (243→32→32→82), which conflicts with the post calling them equal-sized, so interpret that comparison with care.

The edited model is on Hugging Face as RikiyaT/Qwen3.5-0.8B-weight-space-programming. It is based on Qwen/Qwen3.5-0.8B (Apache-2.0) and released under Apache-2.0. The blog's evaluation used a text-only export; the published checkpoint is the full multimodal release, whose language-model tensors were hash-verified to match, and text prompts do not use the vision tower. The demo is the akhaliq/weight-space-programming-demo Space, rebuilt and re-evaluated by Hugging Face's Ahsen Khaliq (the Space page itself has no usage description; you try it from the App tab).

VRAM needs are small. The model card lists about 0.9B parameters, so BF16 weights are roughly 0.9B x 2 bytes, about 1.8 GB, plus KV cache and runtime overhead. This is a general estimate, not an official recommendation. Because the evaluation generates up to 32,768 tokens, a long loop can grow the KV cache by several GB. You can check other models with the VRAM calculator. Keep in mind that the reported numbers come from one seed and one sampling configuration. The model card's independent re-run (all 500 MATH-500 problems x 4 samples, about 51 minutes on an H200) gave 51.0% correct and 31.6% hitting the token limit, but only MATH-500 was re-run; GSM8K and IFEval were not.

The description of ROME/MEMIT is our general understanding; we found no direct comparison with them in the blog. In short, existing methods fix how the update is computed (gradients, closed-form formulas), while here the LLM decides how to update on the spot. The authors list future directions: auditing for backdoors, one-shot initialization, zero-data capability steering and unlearning, grafting circuits between models, and recursive self-improvement. For training-based tuning of the same Qwen3.5 family, see the Qwen3.5-9B fine-tuning guide (in Japanese).

- The gain is mostly loop removal: 93.0% of base outputs hit the token limit, and 33.1% still do after editing. The post itself attributes benefits beyond math to loop removal. It does not show that math reasoning improved
- No gain on Gemma 3 270M: there is no evidence yet that the approach generalizes to other models; editing could not add an ability the model lacked
- Sycophancy detection failed from weights alone: all four attempts failed, and only attention patterns identified all three pairs. It cannot read everything from weights
- Astra itself is not released: redoing the edits requires the gpt-6-astra API, which is expensive (see GPT-6 Astra pricing). The edited model and diffs are public, but there is no guarantee Astra produces the same edit each time
- Evaluation scope: one seed and one sampling setup; the independent re-run covers MATH-500 only. Of the benchmarks evaluated, the table shows only GSM8K, MATH-500, IFEval, and the token-limit rate
- Benchmark contamination is unconfirmed: practice problems came only from training sets, but we could not confirm from primary sources how they relate to Astra's own pretraining data
- Scale limits: networks in experiments 1 to 5 fit in context; larger ones need tools and forward passes. The authors name context length and harness inefficiency as current limits
- The Go comparison: the opponents described as equal-sized are in fact smaller than the network Astra wrote

Since the post, write-ups and community discussion have raised the point that a 7.9% baseline is unusually low for a model this size and that many failures were probably non-answers from loops. The 93.0% token-limit rate is consistent with that reading, but this comes from secondary coverage and discussion, not the primary sources. We could not check the Hacker News thread itself.

This research does not change your workflow today. Astra is expensive, reproduction needs the API, and the results center on one failure mode (looping) in one model. But it shows a few useful directions. First, if a fault is localized to a specific direction, subtracting it can fix the model without retraining, which is a new option for small local models that think without stopping. Second, both the weights and the logs are public, so you can check the method yourself; the logs include each turn's prompt, response, executed code, and evaluation. Third, the authors point to auditing, such as backdoor detection. A practical first step is to try the published edited model on your own prompts and compare token-limit rate and cost. For other models you can run locally, see the local LLM comparison (June 2026) (in Japanese).