Cross Entropy
Chapter 14: Information Theory — Cross-Entropy & Perplexity in LLMs
From the book
Chapter 14: Information Theory. In the chapter mind map this icon labels Cross-Entropy & Perplexity in LLMs. The discussion below is excerpted and lightly edited from § Definition: Cross-Entropy (Discrete) in Mathematics for AI and Machine Learning.
For two discrete probability distributions $p(x)$ and $q(x)$ over the same set of outcomes, the cross-entropy is defined as
What this drawing shows
What you see. Represents the loss comparing a target distribution against predicted probabilities.
In the mind map. Chapter 14 — Cross-Entropy & Perplexity in LLMs. See From the book above for definitions, figures, and worked examples.
Where to read next
Read the full definitions, figures, and worked examples in Chapter 14: Information Theory — see the mind-map node Cross-Entropy & Perplexity in LLMs.