Hardware & sizing

Use an NVIDIA GPU for the supported server. selfjev serve selects CUDA by default and does not offer a CPU or MPS device flag. A full-model Apple Silicon experiment is measured below through the lower-level tree engine; it is not a validated Mac serving setup.

A practical starting point

System RAM and GPU VRAM are separate resources. Extra system RAM does not automatically compensate for too little VRAM in this engine.

Compare measured GPU response times

The interactive hardware comparison shows how long one request takes with SelfJev-4B on A10G, L40S, and H100: texts from 512 to 32,000 tokens, with 1 to 50 questions about the same text. It runs the native tree engine that selfjev serve uses, version 0.4.2, measured on October 5, 2026; the fainter bar in the chart is version 0.4.1, measured on September 30. Each value is the median of up to 20 warmed requests (at least 3), with three answer options per question. The times are measured inside the process, without HTTP or network time. The chart links each GPU to its raw benchmark; the full tables add decisions per second and the cost of each extra question.

Version 0.4.2 against 0.4.1

Version 0.4.2 (October 5, 2026) changed the engine, not the model: the same weights answer faster. One request with one question, median time:

• Long texts gain the most. A 32,000-token text takes 2.7 s on an L40S instead of 9.1 s, and uses less memory (15.5 GB reserved instead of 20.2 GB).
• The answers are the same. Each test set differs by a handful of near-tie questions, none of it significant.
• Many questions about a short text did not get faster. With 25 or 50 questions on a text of 2,048 tokens or less, the A10G and the L40S take about as long as before; the answer branches are most of the work there.

Many questions about one text

The shared-prefix tree reads the text once. Each question then adds only its own short branch. On a 24 GB A10G with version 0.4.2:

• Memory follows the text, not the questions. The weights take about 8 GB, and 50 questions add at most 2.5 GB. Every workload above fits on the 24 GB card.
• Latency grows far more slowly than the number of questions. Fifty questions take 1.4–9.1× as long as one. One request with 50 questions is 5–36× faster than 50 requests with one question each.
• Faster GPUs cut the time, not the memory. Fifty questions about a 2,048-token text take 1,730 ms on the A10G, 804 ms on an L40S and 334 ms on an H100. Memory use is the same on all three.

These requests use synthetic text and are sent one at a time. The 32K rows measure time and memory only: the model was trained on texts up to 16K tokens.

Apple Silicon experiment

We also ran SelfJev-4B on a local M5 Pro with a 20-core GPU and 48 GB of unified memory (September 28, 2026, with the engine before version 0.4.2). The adapter was merged into Qwen3.5-4B in bf16 and scored through TreeServer on PyTorch MPS. The test used synthetic repeated text and three answer options per question. These are local call times, including tokenization but excluding HTTP and network time; each cell is the median of 10 calls after two warm-ups.

The MPS path uses PyTorch's slower reference implementation for the model's gated recurrent operation. The raw samples and benchmark script make this small experiment reproducible. The supported CLI still requires CUDA.

An 8 GB GPU: 4-bit and 8-bit builds

The full model needs about 9 GB for bf16 weights alone, so it does not load on an 8 GB card such as an RTX 4060. Two pre-quantized builds do: 4-bit (NF4) and 8-bit, both made with bitsandbytes from the same merged weights.

pip install "selfjev[serve,gpu,quant]"
selfjev serve --quantized-model Jwuthrich/selfjev-4b-vision-4bit

We scored them on all 3,657 Decision Bench questions on an A10G whose memory we capped at 7.2 GiB, the usable memory of an 8 GB card.

Peak GPU memory and median time for one question with version 0.4.2 (memory in GiB allocated; the weights alone are 8 for bf16 and about 3 for 4-bit):

• Use 4-bit on an 8 GB card. It costs about one point against bf16, reads texts up to 16K tokens, and one 2K-token request takes 454 ms on the A10G (bf16: 434 ms; with version 0.4.1 it took 604 ms). Most of its eval2 loss is in multi-select questions, where the exact set of answers must match.
• 8-bit is slightly more accurate but needs about 10 GB for long texts. It also runs slower (558 ms for the same request).
• Measured on text. Images use the same vision tower as the full model (loaded from the Qwen3.5-4B base on the first image); the quantized path was not scored on images.
• bitsandbytes needs an NVIDIA GPU with CUDA (Linux or Windows). The raw reports are in the repository.

A 16 GB RAM CPU machine is not a supported minimum. No CPU artifact or llama.cpp/GGUF serving path is provided on this page.

Context length and concurrency matter

SelfJev's context limit is configurable with --max-length; the default is 32,768 tokens for the formatted document plus the longest question/candidate path. Prompt formatting and options also consume this budget.

The underlying Qwen3.5-4B configuration has a native 262,144-token context window. That is base-model capacity, not a validated SelfJev serving limit: training used texts up to 16K, and we have not validated full-window inference or accuracy in this engine. Raising the flag alone does not establish usable capacity.

For comparison, Jev's published limits are 32K tokens for state plus the longest question, and 64K tokens for state plus all questions combined (checked September 28, 2026). Self-hosting lets you experiment with a larger budget, subject to memory and validation on your documents.

Memory use grows mainly with document length, as measured above; question and candidate branches add little, and batching adds more. Since version 0.4.2 the text is read without an attention mask; only question and answer tokens carry one, so a request with very many questions and options is what costs extra.

Start with short inputs and conservative batching. The controls are:

uv run --no-sync selfjev serve \
  --max-length 4096 \
  --max-batch-tokens 4096

--max-length limits the state plus the longest question/candidate path. --max-batch-tokens controls packing; it is not a hard memory cap for a single large request tree. Reduce question and candidate counts as well if you run out of memory. Longer-than-allowed input is rejected, not silently truncated.

Disk and first startup

Budget 50 GB of free disk for the checkout, Python/CUDA dependencies, adapter, model cache, and temporary files. This is a planning allowance, not a measured minimum. Building a Docker image with baked-in weights may require additional space.

What has actually been measured?

The current model's latency and memory are measured on A10G, L40S and H100, and a smaller experiment ran on an M5 Pro. The native tree engine also has an A10G correctness check, and the training recipe ran on an L40S. L4 latency, Mac serving and full-model CPU inference remain unvalidated. See the linked reports for the configuration and scope of each measurement.

Sources: question sweep, the 0.4.2 engine report, deployment notes, engine loader, tree packing, GPU engine check, and latency methodology.