Inference
Inference is a trained model doing its job: it takes a request and produces an answer. Unlike training, the weights do not change, and inference is what you pay for when using an AI service.
How it works
A request becomes tokens, passes through every layer, and out comes a probability distribution for the next token. One is chosen, appended to the text, and the whole thing repeats, token by token, until the answer ends.
That is why an answer appears gradually: it is literally being produced piece by piece. And why a long answer costs more than a short one: the work is redone for every token.
A temperature setting controls how the choice is made. Lower means the model more often takes the likeliest option and answers predictably; higher admits less likely ones, which brings variety along with more risk of error.
What drives cost and speed
- Input and output both countYou pay for the context sent and for the answer generated, usually at different rates.
- A long thread gets expensiveThe whole history is resent with every message, so cost climbs as the conversation goes on.
- Model size decidesA smaller model answers faster and cheaper. For simple tasks a large one is overpayment with no gain.
- Context cachingMany providers charge less for an unchanged part of a request that is resent. On long prompts the saving is significant.