Introduction to Large Language Models, ChatGPT, and Generative AI
2025-02-01
Hello world!
- Generative AI, is the biggest data-driven hype of the past years.
Next-word prediction machine
\[P(token_n|token_{n-1}, \cdots, token_1)\]
A token: a single character, a combination of characters, or a word
Next-word prediction machine
\[P(token_n|token_{n-1}, \cdots, token_1)\]
This is nothing new, your phone does something similair:
Byte-Pair Encoding Tokenizer
- Able to:
- encode you whole vocabulary per definition
- Chose precize size you want for your model
- Assign tokens to most important parts of vocabulary
- This does mean that English gets more tokens than Dutch:
Word embeddings
\[ \begin{array}{c@{\hspace{0.3cm}}c@{\hspace{0.3cm}}c@{\hspace{0.3cm}}c@{\hspace{0.3cm}}c} \text{A} & \text{dry} & \text{well!} & \text{Well} & \text{done!} \\ \begin{pmatrix} \phantom{-}0.33\\ -0.51\\ \phantom{-}0.83\\ \phantom{-}0.12 \end{pmatrix} & \begin{pmatrix} \phantom{-}0.97\\ -0.15\\ -0.11\\ \phantom{-}0.85 \end{pmatrix} & \begin{pmatrix} \phantom{-}0.94\\ \phantom{-}0.79\\ -0.34\\ \phantom{-}0.35 \end{pmatrix} & \begin{pmatrix} \phantom{-}0.94\\ \phantom{-}0.79\\ -0.34\\ \phantom{-}0.35 \end{pmatrix} & \begin{pmatrix} -0.02\\ \phantom{-}0.69\\ \phantom{-}0.54\\ -0.12 \end{pmatrix} \end{array} \]
- Dense vectors of Dimension N (hyperparameter of model ~ 728)
- Latent embedding
![]()
- Meaningfull representation
- Encoded semantic information:
King - Man + Woman =
Queen
- These embeddings marked the start of the new NLP era 1
- Generated using shallow networks to:
- Predict middle word from context
- Predict context from middle word
- Tuning via Backpropegation and gradient descent
- This still has some issues…
Word embeddings
\[ \begin{array}{c@{\hspace{0.3cm}}c@{\hspace{0.3cm}}c@{\hspace{0.3cm}}c@{\hspace{0.3cm}}c} \text{A} & \text{dry} & \text{well!} & \text{Well} & \text{done!} \\ \begin{pmatrix} \phantom{-}0.33\\ -0.51\\ \phantom{-}0.83\\ \phantom{-}0.12 \end{pmatrix} & \begin{pmatrix} \phantom{-}0.97\\ -0.15\\ -0.11\\ \phantom{-}0.85 \end{pmatrix} & \color{red}{\begin{pmatrix} \phantom{-}0.94\\ \phantom{-}0.79\\ -0.34\\ \phantom{-}0.35 \end{pmatrix}} & \color{red}{\begin{pmatrix} \phantom{-}0.94\\ \phantom{-}0.79\\ -0.34\\ \phantom{-}0.35 \end{pmatrix}} & \begin{pmatrix} -0.02\\ \phantom{-}0.69\\ \phantom{-}0.54\\ -0.12 \end{pmatrix} \end{array} \]
- Dense vectors
- Latent embedding
![]()
- The numbers are now informative to qualities of the token
- Semanticly-meaningfull:
King - Man + Woman = Queen
- This still causes problems…
What does self-attention look like
What does self-attention look like
\[
Attention \sim Query \cdot Key^{T}
\]
- Conceptual Interpretation:
- Query: I have a Noun, I need a Subject!
- Key: I have a Subject here.
What does self-attention look like
\[
\mathrm{Output\ embedding} \sim \mathrm{Softmax}( Query \cdot Key^{T}) Value
\]
- Conceptual Interpretation:
- Query: I have a Noun, I need a Subject!
- Key: I have a Subject here.
- Data-dependent aggregation of information
Sampling output tokens
- Output is sampled, and therefore a stochastic variable:
- Running the same prompt twice will give 2 different results \[ P_i = \frac{e^{\frac{y_i}{T}}}{\sum_i^n e^{\frac{y_i}{T}}} = \frac{(e^{y_i})^\frac{1}{T}}{\sum_i^n (e^{y_i})^\frac{1}{T}} \]
If we have more compute …
- What if we want to improve our models.
- Companies like OpenAI have more compute available, what should they do?
Optimal model size grows smoothly with the loss target and compute \(\mathrm{budget^1}\)
For optimally compute-efficient training, most of the increase should go towards increased model size1
… we puth it towards the model scale
Models just kept on growing, credit: Julien Simon, Huggingface
GPT3 IS ChatGPT, almost …
- 2020 - Language Models are Few-Shot Learners
- These models are so good at language modeling (SOTA),
that finetuning is no longer needed to perform NLP task - Very strong 0-shot performance in many NLP task (already in GPT2)
- Can perform In-context learning:
- Given a few examples, learn how to perform task
- No model parameters are adjusted at any point
- Emergent property of LLMs
