Hyperparameters and target hardware
The settings every number is computed against.
Two toolbar dialogs set the terms of the analysis. On a window narrower than 1,800 pixels they show as icons: point at one for its name. They matter as much as the architecture: the same model on different hardware, or at a different precision, gives entirely different numbers.
Hyperparameters#
| Group | What you set |
|---|---|
| Model | Width, depth, attention and key-value heads, head dimension, feed-forward width, vocabulary, sequence length |
| Training | Batch size, learning rate, epochs, optimizer, weight decay, warmup, schedule, gradient checkpointing, early stopping |
| Parallelism | Tensor, pipeline and expert parallel degrees, micro-batch size, gradient accumulation, ZeRO stage |
| Data | Dataset size, vocabulary, number of classes |
| Family-specific | Expert count and top-k for MoE, state dimension for SSM, image size for CNN, and so on |
You can also add your own named parameters, for a block that takes something the schema does not list.
Target#
What every metric is computed for. It opens on the machine you are running on — NEURAX detects it rather than asking you to pick from a list — with the memory it has free right now, and what it measured that hardware sustaining.
A machine with no accelerator is a target, not a missing one: the CPU is what the work would run on, and NEURAX sizes against it. The specification database is still there for designing against hardware you do not have — a rented instance, a machine you intend to buy — behind Design for a machine you do not have.
Page 18 of 33. This guide is in English for now. It is the same guide that is built into the NEURAX desktop app (the ? button, or F1). Found a mistake? Tell us.