Every character
has a past.
/* Store K and V in the cache */memcpy(key_buf[l][pos], k, sizeof(k));memcpy(val_buf[l][pos], v, sizeof(v));int ctx_len = pos + 1; 12 /* Scaled dot-product attention logits */float al[BLOCK_SIZE];for (int t = 0; t < ctx_len; t++) { float dot = 0; for (int j = 0; j < HEAD_DIM; j++) dot += q[hs + j] * key_buf[l][t][hs + j]; al[t] = dot * inv_sqrt_d;}Mini GPT.
Nothing hidden.
A character-level transformer, written in a single C file. Every forward pass, gradient, and update is there to follow.
Select a character to see what comes before it. The highlighted loop reads only the available context—future characters stay out of reach.
- Layers
- 02
- Attention heads
- 04
- Context
- 32