1. Part 1 - Tokenization (words to integers) - https://mikexcohen.substack.com/p/llm-breakdown-16-tokenization-words?triedRedirect=true
2. Part 2 - Logits and next-token prediction - https://mikexcohen.substack.com/p/llm-breakdown-26-logits-and-next
3. Part 3 - Embeddings - https://mikexcohen.substack.com/p/llm-breakdown-36-embeddings
4. Part 4 - Transformer outputs (hidden states)- https://mikexcohen.substack.com/p/llm-breakdown-46-transformer-outputs
5. Part 5 - Attention- https://mikexcohen.substack.com/p/llm-breakdown-56-attention?r=658yg&triedRedirect=true

Notes

Embeddings -

https://mikexcohen.substack.com/p/llm-breakdown-36-embeddings

1. Embeddings vectors are part of the LLM, and as such, they are trained through next-token prediction alongside the rest of the model. In this sense, the embeddings vectors are not privileged or special; they are parameters to optimize just like every other parameter in the LLM.
2. After each prediction, the weights are adjusted to increase the model’s next-token prediction accuracy.

Attention

1. • That first term in the softmax function (QK^T/sqrt(d_k)) produces the “raw attention scores.” The idea is that the query vectors encode what each token is searching for, and the keys vectors encode what each token has to offer the query vectors. Imagine a dating app for embeddings vectors: Q is each token’s dating profile and K is the profile of the other tokens in the text. When there’s a good match, their dot product is high; and when there isn’t a good match (meaning the two tokens have no relevant contextual information in common), then the dot product is negative.