Legal SLM · trained from scratch

SLM‑125M

A 125-million-parameter Llama-style language model, pretrained from random weights on a legal and financial corpus. Complete a sentence, step through the model’s next-token distribution, or race one prompt across four temperatures.

125.8M
parameters
16,384
vocab
1,024
context
2.19B
train tokens
10.35
val perplexity
3
epochs
your prompt (a prefix to continue)
ready
The completion will appear here.
ready
ready

Same prompt, same nucleus (top-p 0.95, top-k 50), four samplers that differ only in temperature — the knob that divides the logits before the softmax. Low T concentrates mass on the argmax (safe, repetitive); high T flattens the distribution (varied, eventually incoherent).

What this is base completer

This is a base model, not a chatbot. It was trained on next-token prediction only, so it continues text rather than answering questions. The three tabs are three views of the same object: sampled continuations, the raw next-token distribution those samples are drawn from, and how temperature reshapes it.

The honest quality metric is held-out validation perplexity: 10.35 (lower is better), reached over 3 epochs of pretraining. It speaks the legal register fluently (case-citation phrasing, procedural language) but it does not know facts, at 125M parameters a model holds only about 31MB of usable knowledge. Grounded facts would need retrieval (RAG).

stream 3 datasets clean rule chain dedup + decontaminate 16K byte-level BPE pack 1024-token windows pretrain 8×H100, 3 epochs

Corpus: US case law (~40%), SEC filings (~40%), educational web text (~20%). First call may take ~15–30s while the model wakes from idle.