Executive summary
This project is best understood as a learning archive rather than a single model dump. Each version taught something different: V1 showed how to wire a language model end to end, V2 tightened the training and inference discipline, and V3 proved I could build a full character-level model from scratch with no third-party ML dependencies.
The important result is not “the model is finished.” The important result is that the system now behaves like a real research loop: dataset in, model out, checkpoints saved, previews monitored, and failure modes observable instead of mysterious.
Objective
The original question was simple: how much better can I reason about AI systems when I understand the layers directly instead of treating the model as a black box?
The project intentionally moved from model usage toward model construction. Each version was scoped to force deeper understanding of architecture, data preparation, sampling behavior, and the limits of short-context generation.
Version history
-
V1 · Pre-trained foundation
A PyTorch GPT-style baseline. This version focused on pipeline wiring, tokenization, prompt handling, and the basic mechanics of loading, training, evaluating, and sampling a language model.
-
V2 · Pre-trained refinement
Refactored the project into a more modular structure with cleaner training and inference controls, better evaluation hooks, and role-tagged chat prompting.
-
V3 · Self-trained model
Rebuilt the model in pure Python as a TinyCharRNN with list-based math, atomic JSON checkpoints, and training telemetry so the whole system could be understood from first principles.
-
Why this matters
Moving from pre-trained to self-trained made the trade-off explicit: less raw capability, but much more control, visibility, and intuition about why the model behaves the way it does.
System overview
V3 is a pure Python character language model built around a single recurrent layer. The model stores character embeddings, applies a tanh recurrent update, projects to vocabulary logits, and samples the next character with temperature and top-k filtering.
Input
Model
One hidden state, updated character by character.
Output
Designed for transparency first, not maximum model power.
Training pipeline
V3’s trainer merges the text files in the data directory, builds a character vocabulary, initializes the RNN, warms it up for a throughput estimate, then trains on random contiguous windows of length 128.
This run took roughly 72 consecutive hours on an RTX 3060 Ti with 8 GB of VRAM. I have no romantic explanation for the power draw beyond the obvious: long training runs on consumer hardware are expensive in time, heat, and probably lifespan.
- Hidden size: 256
- Block length: 128 characters
- Base learning rate: 0.03 with step decay
- Checkpoint cadence: every 1,000 steps
- Preview cadence: every 1,000 steps
- Gradient handling: manual clipping to keep recurrent training stable
The gap is real progress, but not enough for clean semantic control.
Inference behavior
The inference path is intentionally short. The server loads the saved weights, primes the model with the user prompt, then generates text using temperature and top-k sampling.
What matters is the shape of the output. Early versions mostly produced noise. By late V3, the model could reliably form character-level patterns, line breaks, repeated speaker markers, and fragments of dialogue-like structure. That is a meaningful sign of learning, even though the generated content still drifts and loses semantic continuity.
- Runtime seed: raw user input or role-framed prompts
- Default generation settings: temperature 0.8, top-k 50
- Prime window: last 64 characters during warmup
- Observed behavior: syntax and rhythm before meaning
V3 outcomes
V3 does not yet produce consistently coherent responses. It does, however, learn sentence-like structure, speaker cadence, and local grammatical patterning from the corpus.
- Strength: clear structural rhythm and grammar-like form from training data.
- Current gap: semantic coherence, intent tracking, and sustained logical consistency.
- Key takeaway: the model can mimic form before it masters meaning.
- Practical win: the entire failure mode is now inspectable instead of hidden behind a library.
What I learned
Building the progression made model behavior easier to debug and discuss. Instead of treating output as magic, I can now reason about likely causes in data, training choices, and architecture constraints.
The biggest learning was that capability does not appear all at once. First the model learns token rhythm, then local structure, then weak role imitation, and only later any semblance of long-form consistency. That ordering changed how I think about every AI system I build.
What I would do next
- Add a stronger prompt template in the chat server so the model is always primed with a stable system role.
- Separate corpora cleanly so training data is reproducible and each experiment has a single purpose.
- Add repetition guards and simple decoding penalties to reduce looping and drift.
- Track validation perplexity and sample quality together so the model is judged by more than just loss.
- Keep the pure-Python implementation, because the value of V3 is understanding every moving piece.
Links
Note · ARES is intentionally experimental. The point is not just the output quality, but the fact that the full stack is now legible, measurable, and mine.