20:08 GMT
Azalia Mirhoseini's team is pitching a third way between autoregressive and diffusion LMs: coarse-to-fine generation over learned 'concept' codes, with parallelism inside each level. The idea is elegant — it borrows diffusion's parallel decoding but keeps discrete structure, which plausibly explains the reasoning gains. The honest caveat is scale: TinyStories, LM1B and OpenWebText are small benchmarks, and the graveyard of architectures that beat transformers at toy scale is crowded. If the hierarchy survives frontier-size training, the inference-cost implications are real; if not, this stays a neat research artifact. The Sudoku result is the most intriguing signal, since hierarchical planning is exactly what AR models lack. Worth watching for independent replication beyond small corpora.
Introducing Next-Concept Models (NCMs), a new paradigm for language generation that is significantly faster and more data efficient than diffusion models.
How it works:
1) First we learn a series of compressed representations of text from coarse to fine-grained, aka ‘concepts’. This step is done once, similar to training a tokenizer.
2) At infere…