Terms You'll See in Every Experiment

The Model

Qwen3.6 35B-A3B turns text into tokens, then repeatedly transforms those tokens through 40 layers before producing probabilities for the next token.

text
  ↓
tokenizer
  ↓
token IDs
  ↓
embedding
  ↓
40 layers
  ↓
LM head
  ↓
logits
  ↓
next token

Token

A token is a piece of text. It is not necessarily a whole word.

"Hello world" → ["Hello", " world"]  = 2 tokens
"transformer" → ["transform", "er"]    = 2 tokens
"I"            → ["I"]                  = 1 token

The model vocabulary contains 248,320 token IDs.

Layer

Qwen3.6 35B-A3B has 40 layers. During autoregressive decoding, each newly generated token passes through all 40 layers in order.

Token
  ↓
Layer 0
  ↓
Layer 1
  ↓
...
  ↓
Layer 39
  ↓
next-token logits

The architecture is organized as 10 groups of four layers: three GatedDeltaNet layers followed by one attention layer.

GatedDeltaNet

GatedDeltaNet is a recurrent-style sequence-mixing mechanism. Instead of standard attention over the full history, it maintains a compact state that is updated as tokens arrive.

new token
   │
   ▼
GatedDeltaNet
   │
   ├── read recurrent state
   ├── compute update
   └── write new state
   │
   ▼
next layer

Attention

The attention layers let the current token retrieve information from previous tokens. During decoding, each attention layer maintains its own KV cache.

current token
     │
     ├── Q ───────────────┐
     │                    │
     ├── K ──► KV cache ──┤
     │                    │
     └── V ──► KV cache ──┘
              │
              ▼
          attention
              │
              ▼
          new hidden state

The important distinction: the entire KV cache is not passed through all 40 layers again. Each attention layer reads its own cached keys and values and appends the new token's K/V.

MoE — Mixture of Experts

The feed-forward part is a Mixture of Experts. Each MoE layer contains 256 routed experts. A router scores the experts for the current token and selects 8 routed experts.

hidden state
     │
     ▼
   Router
     │
     ├── Expert 12  ──┐
     ├── Expert 47  ──┤
     ├── Expert 89  ──┤
     ├── ...          ├──► combine
     └── 8 total ─────┘

256 experts exist.
Only 8 routed experts are selected for this token.

Expert

Each expert is a small feed-forward network with three learned weight matrices.

hidden (2048)
     │
     ├── gate projection ──┐
     │                     │
     ├── up projection ────┤ → gated activation
     │                     │
     └─────────────────────┘
                    │
                    ▼
              down projection
                    │
                    ▼
             expert output

For the configuration we're using, an expert has an intermediate dimension of 512, so its three matrices contain roughly 3.1M parameters.

Tensor

A tensor is simply a multi-dimensional array of numbers. Model weights, hidden states, router outputs and caches are represented as tensors.

1D: [1.2, 3.4, 5.6]
2D: [[1.2, 3.4], [5.6, 7.8]]
3D: [[[...], [...]], [[...], [...]]]

Weight

A weight is a learned number. During inference, weights are read but not updated.

GGUF

GGUF is the model container we will initially support in inferedThoughts. It contains model metadata plus tensor metadata and the binary tensor data.

GGUF
├── header
├── metadata
├── tensor directory
│   ├── name
│   ├── shape
│   ├── type
│   └── offset
└── tensor data

Quantization

Quantization stores weights using fewer bits than their original floating-point representation. This reduces the amount of data required to store and move the model, with some approximation error.

Our current GGUF uses IQ4_XS and Q5_K tensor types.