InferedThoughts · Model Execution

One Token Through Qwen3.6

August 29, 2026 Qwen3.6 35B-A3B V0 Architecture
A concrete walkthrough of what happens to one generated token: hidden state, router scores, expert selection, DeltaNet state, attention KV cache, and the final logits.

Start With One Token

During decoding, we already have a sequence of previous tokens in the model's state. A new token arrives and must be processed through the network to produce the next token.

previous context
       │
       ▼
   new token ID
       │
       ▼
   embedding
       │
       ▼
   hidden state

The Hidden State

The embedding becomes a hidden vector of width 2048. This vector is the main piece of information passed from one layer to the next.

hidden state
[ h₀, h₁, h₂, ... h₂₀₄₇ ]

shape ≈ [2048]

One Layer

Each layer receives the hidden state and transforms it. The repeating architecture contains three GatedDeltaNet layers followed by one attention layer in each four-layer group, with the MoE feed-forward computation associated with the layer.

                 hidden state
                      │
                      ▼
              ┌───────────────┐
              │   Layer N     │
              │               │
              │ DeltaNet OR   │
              │ Attention     │
              │               │
              │     +         │
              │     MoE       │
              └───────┬───────┘
                      │
                      ▼
                hidden state'

The Router

At an MoE layer, the router looks at the current hidden state and produces scores for the 256 experts.

hidden state
     │
     ▼
 router projection
     │
     ▼
256 scores
     │
     ▼
top 8
     │
     ├────────► expert A
     ├────────► expert B
     ├────────► ...
     └────────► expert H

The router does not run all 256 experts. It determines which experts should receive this token.

Inside One Expert

Each selected expert transforms the same input hidden state using its own learned weights.

                 hidden [2048]
                      │
            ┌─────────┴─────────┐
            ▼                   ▼
       gate_proj             up_proj
            │                   │
            └───────┬───────────┘
                    ▼
              gated activation
                    │
                    ▼
               down_proj
                    │
                    ▼
               expert output

Each expert has three weight matrices. With intermediate dimension 512, one expert contains roughly 3.1M parameters.

When the Layer Is Attention

Attention is different from GatedDeltaNet because it explicitly retrieves information from previous tokens using the layer's KV cache.

                  hidden state
                       │
             ┌─────────┼─────────┐
             ▼         ▼         ▼
             Q         K         V
             │         │         │
             │         └────┬────┘
             │              ▼
             │          KV cache
             │              │
             └──────┬───────┘
                    ▼
                attention
                    │
                    ▼
              updated state

KV Cache

For every attention layer, previous tokens' keys and values are retained so that decoding does not have to recompute them from scratch.

Layer N KV cache

token 0 → K₀ V₀
token 1 → K₁ V₁
token 2 → K₂ V₂
...
token T → Kₜ Vₜ

new token:
        compute Kₜ₊₁ Vₜ₊₁
                  │
                  ▼
             append cache

Each layer has its own cache. A new token reads the cache belonging to that layer when it reaches the attention operation.

When the Layer Is GatedDeltaNet

GatedDeltaNet maintains a recurrent state instead of using the standard attention KV-cache mechanism.

previous DeltaNet state
          │
          ▼
      current token
          │
          ▼
     GatedDeltaNet
          │
          ├── read state
          ├── compute update
          └── write state
          │
          ▼
     new DeltaNet state

Where the Weights Live

For V0, we deliberately don't optimize placement. Conceptually, however, the model has a storage hierarchy:

GGUF on SSD
     │
     ▼
system RAM
     │
     ▼
GPU VRAM
     │
     ▼
CUDA kernels

Later versions of inferedThoughts can make this hierarchy explicit. V0 simply needs to load the required tensors and produce correct results.

One Complete Decode Step

new token
   │
   ▼
embedding
   │
   ▼
┌─────────────────────────────┐
│ Layer 0                     │
│   DeltaNet + MoE            │
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Layer 1                     │
│   DeltaNet + MoE            │
└──────────────┬──────────────┘
               ▼
              ...
               ▼
┌─────────────────────────────┐
│ Layer 39                    │
│   Attention + MoE           │
│   + KV cache access         │
└──────────────┬──────────────┘
               ▼
             norm
               │
               ▼
            LM head
               │
               ▼
            logits
               │
               ▼
            sampler
               │
               ▼
          next token

The key distinction for our runtime: the hidden state travels through every layer for every generated token. The KV cache does not travel through all layers; each attention layer reads and updates its own cache. MoE routing determines which expert weights are actually needed at that layer.