Context Window
The context window is how much text a model holds at once: your request, the whole prior conversation and the answer. It is measured in tokens, and anything outside the window stops existing for the model.
How it works
A model remembers no past conversations. Every request is sent along with the whole history, and that history is the "memory". Once the total exceeds the window, the oldest part is dropped or compressed.
The window is shared between input and output. Load a document filling 90% of it and you leave the model little room to answer, which comes out shorter than you expected.
A large window does not mean good use of it. Models consistently handle the start and end of a long text better than the middle, so filling the window "just in case" usually hurts quality.
How to work with it
- Put what matters at the edgesAn instruction at the very start plus a restatement of the key requirement at the end beats the same words in the middle.
- Less text, better chosenThree relevant pages produce a better answer than three hundred with those three hidden inside.
- A long thread gets expensiveThe history is resent in full every time, so the cost per message climbs as the conversation goes on.
- For large volumes there is RAGRather than loading a whole corpus into the window, retrieve the relevant pieces and pass only those.