This is a rich topic. Let me build an interactive explorer showing the classic token embedding axes — like those from Word2Vec — and what happens when you rotate them to find human-interpretable dimensions.The core idea behind each tab is what makes this conceptually interesting:

High-dimensional embeddings aren't interpretable by default. A typical embedding might be 768 or 1536 dimensions, and each raw axis is just a weight from training — not "gender" or "sentiment." But the geometry of the space develops meaningful structure on its own.

How interpretable axes are found:

• You take pairs of known contrasting words (man/woman, good/bad, formal/informal) and compute the difference vector
• That difference vector often generalizes — it's a direction in the space, not tied to those specific words
• You can then project any token onto that direction to score it along that axis

What this visualization simplifies:

• Real embeddings are hundreds of dimensions compressed to 2D here, so positions are approximate/illustrative
• The axes aren't perfectly orthogonal in practice — gender and royalty correlate slightly in any corpus trained on historical text
• Modern LLM embeddings (transformer-based) are richer and more contextual than Word2Vec, making clean linear directions harder to isolate — though they still exist

Why this matters: This is the basis of techniques like RLHF steering vectors, activation patching, and concept probing — you can find the "has a negative connotation" direction in a model's residual stream and push it around to understand or control model behavior.