Skip to content
NEURAX
Download

Measurement report

What we measured, and what we did not

A reproducible check of NEURAX 0.24.5 across data reading, architecture analysis, generated PyTorch, studio tests, a real training run and exported weights. These are scoped test results, not a single score for every model or dataset.

Measured 7 October 2026 · NEURAX 0.24.5

There is no honest overall accuracy percentage

Parameter counts and generated-model consistency can be checked against known values. Runtime and memory are estimates tied to hardware. Predictive quality can only be measured on a particular task and held-out data. Combining these into one percentage would hide what each result actually means.

100%
Parameter-count agreement

80 / 80 templates · compiler and generated PyTorch

87.5%
Templates passing runtime checks

21 / 24 · 3 safely stopped at 120 seconds

100%
Frontend test pass rate

1,015 / 1,015 executed · 5 skipped

Results by stage

  1. Stage 01

    Dataset reading

    53 targeted tests passed

    The data-source parser tests passed for the formats and edge cases they cover. This is not proof that every arbitrary client folder or label scheme is understood. Training data should be prepared and labeled before import; this audit used test fixtures, not customer data.

  2. Stage 02

    Architecture counts

    8 published references · 3 tests passed

    Reference models are compared with rounded published parameter totals. Allowed differences are 1–10%, depending on the reference. This checks parameter formulas; it does not measure model quality.

  3. Stage 03

    Compiler to PyTorch

    80 / 80 templates · 0 count mismatches

    The service compiled all 80 current templates, generated their PyTorch code, and counted the generated parameters on PyTorch's meta device. Every count matched the compiler exactly, including models up to 671 billion parameters. This proves internal count consistency, not successful execution at that scale.

  4. Stage 04

    Executable model check

    21 / 24 passed · 3 safely stopped

    On this CPU, 21 eligible templates completed a forward pass, backward pass and optimizer step. Two Mamba templates and one Stable Diffusion template exceeded the 120-second checker limit; the checker stopped them and confirmed that no training started. The remaining 56 templates were outside this runtime smoke check's size or input-kind scope.

  5. Stage 05

    Decision approval gate

    8 frontend tests + 22 agent tests passed

    The focused tests verify that training code is not written and a run is not started before approval, and that a checked approved run starts once. These are automated tests of the gate, not a manual audit of every Decision screen.

  6. Stage 06

    Training to weights

    1 synthetic run · 12 / 12 steps · 212 parameters

    A real two-epoch training process finished, recorded finite losses for all 12 steps, and wrote a non-empty export/last.safetensors file. This verifies artifact production on one small fixture; it does not show that the model learned a useful real-world task.

  7. Stage 07

    Using exported weights

    8 predictor tests passed

    Automated tests cover loading and prediction from exported files, including a Safetensors-only export without the .pt file. They use fixtures and do not certify every generated architecture or a hosted API deployment.

  8. Stage 08

    Studio interface

    1,015 passed · 5 skipped · 1 file skipped

    The full frontend suite ran 129 test files. It covers the interface behavior represented by those tests; it is not a manual inspection of every screen, operating system or user journey.

Test conditions

Linux test host: Intel Core i5-8365U, 4 cores / 8 threads, 23 GiB RAM. About 3.4 GiB was available when the host was first measured. PyTorch CPU backend. The 80-template compiler and generated-model sweep took about 18 minutes 51 seconds. These results describe this host and this software version, not every client's computer.

What remains unmeasured here

  • There is no measured global percentage for the agent's understanding of every prompt.
  • Prediction quality on customer tasks and named public datasets was not benchmarked in this audit.
  • Prediction-time and memory forecasts were not compared with completed runs across a hardware cohort here.
  • A full browser-driven audit of every screen and every platform was not performed.
  • A Safetensors file is a local model artifact. A deployed network API was not tested by this audit.
  • Training is still in active development and is not yet stable. A supported run can continue from an earlier NEURAX export; the Studio does not currently fine-tune Hugging Face or ModelScope checkpoint weights.

Method: the counts above come from the named automated test suites and one disposable synthetic training fixture. Exact compiler/generated counts are reported separately from runtime checks, expected safe refusals, estimates and task-level quality. A passing test means only that its stated assertions passed.