What the numbers mean
The figures that decide whether a design is viable.
The analysis panel on the right has three tabs — Architecture, Performance and Hardware. Between them they show the figures below, each with where it comes from: exact, measured on your machine, estimated, or unknown with what is missing.
Model#
| Metric | Meaning |
|---|---|
| Parameters | Total learnable weights — the headline size. |
| Layers | Depth, taken from the layer stack, not from how many blocks you drew. |
| Active per token | For a mixture of experts, the parameters one token passes through. |
Memory#
| Metric | Meaning |
|---|---|
| Peak memory | Weights, gradients, optimizer states and kept activations, per device. This decides whether the model fits. |
| Weights | Parameters at the chosen precision; mixed-precision training keeps them in float32. |
| Kept activations | The tensors each operation keeps for the backward pass. Estimated; activation checkpointing trades them for recomputation. |
| Gradients | One value per parameter. |
| Optimizer state | AdamW keeps two more values per parameter; Adafactor a row and a column per matrix. |
| Largest batch that fits | The largest batch whose peak fits the free memory, found by compiling each candidate. |
Compute#
| Metric | Meaning |
|---|---|
| Step FLOPs | One optimizer step, forward and backward, counted the way PyTorch counts them. |
| FLOPs per token | Forward FLOPs per token, the usual scale for comparing language models. |
| Limited by | Whether arithmetic or moving memory takes most of the step. |
Performance and cost#
| Metric | Meaning |
|---|---|
| Step time | One optimizer step: FLOPs over the machine's throughput, element-wise work over its memory bandwidth. |
| Throughput | Tokens or samples per second at the configured batch size. |
| Training time | Steps from the dataset size and the epochs, times the step time. |
| Training cost | Device hours times an hourly price; unknown when no price was given. |
| Energy / CO₂ | From the power draw and the grid's carbon intensity; unknown when either is missing. |
Page 29 of 33. This guide is in English for now. It is the same guide that is built into the NEURAX desktop app (the ? button, or F1). Found a mistake? Tell us.