Model architecture, training, and runtime

Basic AI/ML familiarity helps, because this document uses terms like hidden size, block length, learning rate, checkpoint cadence Checkpoint cadence
How often the model saves its progress to disk — here every 1,000 steps. Like auto-saving a video game so a crash never costs you much. , preview cadence Preview cadence
How often it pauses to print a sample of its current writing — here every 1,000 steps — so you can watch it slowly improve. , gradient clipping Gradient clipping
A safety cap for updates during training. If a math step becomes too large, clipping limits it so learning stays stable instead of blowing up. , and perplexity Perplexity
A score for how surprised the model is by real text. Lower is better, because it means the model predicts the next character or word more confidently. . You are still welcome to read it without that background; the point of the document is to explain what those terms mean in the context of the build, not to gatekeep the project.

The point of V3 was not to beat a modern foundation model. The point was to own the full stack of a language model experiment: tokenizerTokenizer
The translator between text and numbers. It gives every character its own ID number going in, and turns the numbers back into characters coming out., data assembly, recurrent architecture, BPTTBPTT
Short for Backpropagation Through Time. The network replays the sequence step by step, sees where it guessed wrong, and passes that error backwards so it can fix its settings., sampling, checkpointing, server wrapper, and performance monitoring. Once you own all of that, the output is easier to interpret because the moving parts are no longer hidden behind a framework abstraction.

V3 was the first version where the behavior of the system could be traced all the way from a raw character token through a hidden state update to the final sampled character. That matters because it changes the debugging question from “why is the model weird?” to “which part of the pipeline is responsible for the weirdness?”

V3 is a one-layer Elman RNNOne-layer Elman RNN
A simple neural network that reads text one character at a time and keeps a tiny 'memory' of what came just before. 'One-layer' means the most basic, single-step version.. At runtime, each input character is converted into an embedding vector, combined with the previous hidden state, passed through tanh, and projected to a vocabulary-sized probability vector. There is no attention, no multi-head machinery, and no token-level transformer stack. It is intentionally smaller and easier to reason about.

h_t = tanh(E[x_t] + Whh_T @ h_prev + b_h)
y_t = softmax(Why_T @ h_t + b_y)

def _step(self, idx, h_prev):
    x = self.E[idx][:]
    Whh_h = vecTmat(h_prev, self.Whh)
    pre = [xi + wi + bi for xi, wi, bi in zip(x, Whh_h, self.bh)]
    h = tanh(pre)
    logits = vecTmat(h, self.Why)
    for j in range(len(logits)):
        logits[j] += self.by[j]
    probs = softmax(logits)
    return h, probs

Here is the whole system in plain language, no background assumed. Imagine teaching someone to write by covering a book one letter at a time and quizzing them on the very next letter, over and over, thousands of times. That is basically what ARES V3 does to itself until it starts to sound right.

1. Feed it text. The model reads a big pile of training text (the corpusCorpus
   Just a big pile of training text. Here it is mostly Shakespeare, from a file called tiny_shakespeare.txt.), which is mostly Shakespeare from a file called tiny_shakespeare.txt.
2. Turn letters into numbers. The tokenizer gives every character its own ID number, because the math can only work on numbers, not letters.
3. Read one character at a time. The network (it is called TinyCharRNN) takes each character and updates a small hidden state — a little memory of what it has seen so far.
4. Guess the next character. From that memory it produces a chance for every possible next character, then picks one of them.
5. Check the guess. While training, it compares its guess to the real next character and measures how wrong it was (the loss).
6. Learn from the mistake. Using BPTT it works out which settings caused the error (the gradientsGradients
   Numbers that point each setting toward fewer mistakes — basically 'which way is downhill' toward better guesses.), clips them so nothing explodes, and nudges every setting a tiny bit toward a better guess.
7. Repeat about 10,000 times. Every 1,000 steps it saves a checkpoint and prints a short preview, so you can literally watch it improve.
8. Chat with it. A tiny built-in standard-library Python HTTP server (app.py) lets you type a message in the browser; the model is seeded with your text and writes back one character at a time.

text  -->  tokenizer  -->  numbers  -->  RNN (+ memory)  -->  guess next char
                                              ^                         |
                                              +------ nudge weights -----+
                                                  (training only)

The V3 trainer merges text files from the data directory, joins them into one corpus, and builds a character tokenizer from that combined text. That is simple on purpose. The model is character level, so the tokenizer is not trying to be clever; it just maps characters to integer IDs and back.

That simplicity is a feature. It removes one more source of magic. If the model outputs garbage, the issue is probably not a complicated subword merge scheme or a missing Hugging Face utility. It is likely the corpus, the context length, the learning dynamics, or the sampling path.

The whole tokenizer really is this small. It sorts the unique characters, hands each one an ID, and keeps two lookup tables so it can go from text to numbers and back. That is the entire thing:

class CharTokenizer:
    def __init__(self, text):
        chars = sorted(list(set(text)))          # every unique character, in order
        self.stoi = {ch: i for i, ch in enumerate(chars)}   # char  -> number
        self.itos = {i: ch for ch, i in self.stoi.items()}  # number -> char

    def encode(self, text):  return [self.stoi[c] for c in text if c in self.stoi]
    def decode(self, tokens): return ''.join(self.itos.get(t, '?') for t in tokens)

The training loop samples random contiguous windows of length 128 and runs truncated backpropagation through time (BPTT) BPTT
Short for Backpropagation Through Time. For an RNN, it means unfolding the sequence step by step, measuring mistakes at each step, then sending that error backward so the model can adjust its weights. . The hidden size is 256. Learning rate starts at 0.03 and decays by half every 10,000 steps. Gradients are clipped manually so the recurrent update does not blow up during long runs.

There is also a warmup phase that exists only to estimate throughput before the real loop starts. That is useful because the project is expensive enough that knowing the step rate matters. It gives you a realistic ETA instead of a fantasy number.

for step in range(start_step, TOTAL_STEPS + 1):
    model.lr = BASE_LR * (0.5 ** (step // 10000))
    s = random.randint(0, len(ids) - BLOCK_LEN - 2)
    x = ids[s:s + BLOCK_LEN]
    y = ids[s + 1:s + BLOCK_LEN + 1]
    loss = model.train_step(x, y)
    if step % SAMPLE_EVERY == 0:
            preview = model.generate(tok, "ROMEO:\n", 200, PREVIEW_TEMP, PREVIEW_TOPK)

Backpropagation through time Backpropagation
The core training method for neural networks. It calculates how much each weight contributed to the model's error, then nudges those weights in the direction that reduces future errors. is where the model stops being a toy. Every time step contributes to the gradients, and the recurrence means errors can amplify quickly. That is why clipping matters so much here. If you remove clipping, the model becomes noticeably less stable during long sessions.

The pure-Python implementation makes this visible instead of abstract. You can literally see the gradient buffers being assembled, clipped, and applied. There is no hidden optimizer magic doing the bookkeeping in the background. That is slower, but it is the whole reason the project teaches well.

clip_mat(dWhy, 0.25)
clip_mat(dWhh, 0.25)
clip_mat(dE, 0.25)
clip_vec(dbh, 0.25)
clip_vec(dby, 0.25)

eta = self.lr
for i in range(self.hidden):
    for j in range(self.vocab_size):
            self.Why[i][j] -= eta * dWhy[i][j]

And the “math by hand” claim is literal. Even the famous building blocks — softmax, tanh, and cross-entropy — are just a few lines each in mymath.py, with no library doing the work underneath:

def softmax(logits):                 # turn raw scores into probabilities that add to 1
    m = max(logits)
    exps = [math.exp(x - m) for x in logits]
    s = sum(exps)
    return [e / s for e in exps]

def cross_entropy(probs, target_idx, eps=1e-9):   # how wrong was the guess? (lower = better)
    return -math.log(probs[target_idx] + eps)

def clip_vec(v, th=1.0):             # the safety cap that stops updates from exploding
    for i, x in enumerate(v):
        if x >  th: v[i] =  th
        if x < -th: v[i] = -th

Checkpoints are written atomically. That choice sounds boring until you interrupt a long run and a partial file would otherwise brick the resume path. V3 stores per-step snapshots, a latest model copy, and a pointer file that says which snapshot is considered current.

The resume logic is intentionally defensive. It tries the pointer first, then falls back to the most recent step file, then starts from scratch if necessary. That is the kind of boring reliability you only appreciate after a 72-hour run.

Inference is seeded directly from the user prompt. The browser chat endpoint takes a message, wraps it in a simple generation call, and samples character by character. The warmup on the seed text is what makes the model feel like it is continuing a conversation rather than blindly starting over.

The practical effect is that the model learns form before meaning. You get speaker labels, line breaks, and Shakespeare-like cadence first. Semantic control, consistent intent tracking, and instruction-following do not emerge reliably in this run.

reply = model.generate(
    tokenizer,
    seed=msg,
    max_new=160,
    temperature=0.8,
    top_k=50
)

The loss curve matters, but it does not tell the whole story. The model can improve numerically while still sounding like a deranged telegram machine. That said, the trend is still useful because it shows the system moving from pure noise toward stable local structure.

The published training log records a completed run through 10,000 steps. Higher values can be configured, but this document reports the 10,000-step run as the completed public baseline.

Sample progression:

Step 1:
ROMEO:
of-LelW5sjdb+'Puurab xVl :b6hVlCd3LzRIrRao6CESbLNp;HqCWW

Step 8,000:
ROMEO:
USER: Deasas.
ARES: yowust iwact.
USER: Dead dang geakacing.
ARES: Woinel is ft?

Step 10,000:
ROMEO:
USER: hraerarg doug?
ARES:  rer le.
ARES: thhe Poure  ulmolm lere...

This is the part that turns the project from a static model into a learner. Every pass through train.py follows the same rhythm: grab a chunk of text, predict the next character at every position, measure the mistake, then adjust the network a tiny amount so it does slightly better next time.

1. Pick a random window. The trainer chooses 128 consecutive character IDs from the dataset, so each update sees a small stretch of real text.
2. Shift the answer by one. The input is characters 1..128 and the target is characters 2..129, which teaches the model to predict the next character instead of copying the current one.
3. Run the forward pass. TinyCharRNN reads the sequence, updates its hidden state, and produces a probability for each possible next character.
4. Measure the loss. Cross-entropy punishes the model when it assigns low probability to the correct next character.
5. Backprop through time. The errors are replayed backward across the sequence so earlier characters also get credit or blame for the final result.
6. Clip the gradients. If the math gets too large, the values are capped so the training run does not explode.
7. Update the weights. The network nudges the embeddings, recurrent weights, output weights, and biases a little bit toward better predictions.

The important bit is scale. It is not trying to learn everything in one move. It learns by making thousands of tiny corrections, which is why a run can take hours or days and still produce visible improvement only in pieces.

When the model is chatting, it is not always choosing the single most likely next character. It can also be told to act a little safer or a little more creative. That is what the temperature and top-k settings do.

In plain English: temperature changes the mood of the guess, and top-k changes how many options are even allowed to compete. Together they decide whether the result feels careful, messy, or somewhere in between.

reply = model.generate(
    tokenizer,
    seed=msg,
    max_new=160,
    temperature=0.8,
    top_k=50
)

This is where the project stays honest. A custom RNN like this can learn patterns, rhythm, and local structure, but it still has serious limits. It is not pretending to be a giant language model, and the results show that clearly.

• It only remembers a short context window, so long-range consistency is weak.
• It works at the character level, so it learns spelling and punctuation before real meaning.
• It can sound convincing for a few lines and then drift or repeat itself.
• Its quality depends heavily on the training corpus and the exact checkpoint it resumed from.
• It is slow compared with modern library-based models because everything is written in pure Python.

The long run time is not theoretical. The published 10,000-step run took roughly 72 consecutive hours on a CPU. The implementation performs its numerical operations directly with Python lists and the standard library, so training is computationally expensive in wall-clock time even for a small model.

I did not preserve the exact CPU model for that run, so this page keeps the timing and avoids pretending to have a more specific hardware record than the logs support. This section exists because a realistic cost model is part of the engineering story, not a footnote.

The model learned form faster than meaning. That is the big takeaway. It can imitate rhythm, punctuation, and speaker structure before it can sustain a coherent thought over long spans. That is why the samples sometimes look like a conversation but still fall apart when you ask for sustained reasoning.

• It can learn short-range character statistics.
• It can learn repeated prompt shapes and response cadence.
• It can learn to look conversational without actually being consistent.
• It is still limited by the short context window and the simplicity of the recurrent cell.

• Add stronger prompt scaffolding in the chat server.
• Separate datasets into explicit experiment sets.
• Add repetition penalties or no-repeat constraints in sampling.
• Track validation perplexity alongside loss and sample quality.
• Build a side-by-side checkpoint viewer so sample drift is easy to inspect.
• Keep the pure-Python core, because the educational value is in the transparency.