What Happened

The answer a memory hierarchy disguised as a 30B Transformer.

Meta pitches Muse Glimmer device: autonomous, multimodal, no cloud required. engineering problem a product claim: fit a capable 30B-class model, a long working history, and a perception stack into consumer hardware. The answer a memory hierarchy disguised as a 30B Transformer.

direct : Muse Glimmer is “purpose-built for autonomous agentic tasks on consumer hardware,” “without requiring cloud infrastructure or network access.” The promise is demanding because an agent's workload is long-lived. Hours of history and tool transcripts stay resident, screenshots and documents get reread mid-task, inside the 24 or 32 GB envelopes quantized releases.

Muse Glimmer is a roughly 30-billion-parameter, decoder-only multimodal model: a vision encoder, a projector, and a dense language model. In BF16 the checkpoint weighs about 55 GiB, which would overflow envelopes before a single token of context, answer : Meta ships roughly four-bit quantized variants language model below 20 GB. The compressed a 131,072-token context, a resident vision tower, and a speculative-decoding drafter, smaller when the language model does. answer is architectural: spends memory, information each layer carries.

Why It Matters

Muse Glimmer is built around a deliberate division of labor. In most layers, attention is local: positioned bounded to a 2,048-token window. In every fourth layer, attention entire context , retrieving primarily by content. Only the attention alternates; identical. Thirty- provide a retrieval behaviors, / stored . On the visual side, a large ViT performs expensive perception once, compresses neighboring patches four-to-one, result to the language decoder as ordinary tokens.

Per-sequence design, so nearly all the memory a running instance 's parameters. weight quantization unusually well here. weights are compressed, the freed memory can be turned into longer contexts, larger batches, a resident perception tower, or a speculative-decoding drafter.

Because Muse Glimmer is dense, every generated token passes through all 52 text blocks. routed experts waiting unused in memory. This gives predictable execution, decoding heavily dependent on repeatedly reading a weights.

There are therefore 39 sliding-attention layers and 13 full-attention layers. The local window is 2,048 tokens.

What Comes Next

A position i can directly recent interval ending at i. But local receptive fields compound with depth. Ignoring boundary effects, three stacked causal windows expose a token indirectly to roughly 1 + 3 × (2048 − 1) = 6,142 positions: 6,141 predecessors itself. The global layer that follows does not receive raw isolated tokens; it receives representations that already summarize several thousand tokens of ordered local structure.

-layer cycle: builds immediate lexical and syntactic relationships, combine them into progressively larger local structures, and the closing full layer retrieves relevant summaries from anywhere in the context.

The division is soft, of course. Local layers carry global information forward in their residual streams, and global layers can attend locally. Still, the masks impose a strong prior: most computation refines nearby structure, while occasional layers handle long-range communication.

cheaper than making all 52 layers global, especially . Long-context compute is another matter: during prefill, the 13 full-attention layers still do quadratic attention sequence length. FlashAttention-like kernels avoid materializing the full attention matrix, products. Muse Glimmer makes 131K context memory-feasible; a 131K prefill equivalent to a 4K prefill.

First, a reminder layers are giving up: RoPE rotates , and because the two rotations compose when the vectors are scored against each other, the attention depending relative displacement j−i. That composition is how a Transformer normally feels distance.

Muse Glimmer θ=500,000 . -attention layer, however, the per-, and the implementation passes no positional embedding into attention - the configuration called NoPE. The full layers therefore score Q–K compatibility without a direct rotary position term.

Order still reaches these layers through two doors. First, the causal mask tells position i positions at or before i. Second, every global already passed through three RoPE-equipped local layers. A vector representing the word “bank” near “river” is different from one representing “bank” near “loan,” and both vectors encode produced them. Later global layers also receive residual states modified by earlier global layers.

Explore more: Software & AI Guide