Definition: inference

The part of an AI chatbot or agent that generates an answer for the user. A peculiar name for sure; however, the term dates back to very early AI (see expert system). The hardware that inference software uses is typically in a data center but can be in the user's own computer. Increasingly, inference operates in a desktop computer.

Also called "inference runtime" and "inference engine," inference software uses the statistical patterns in an AI language model for its knowledge. It is the inference software that generates the running dialogue between the user and the computer. See What can AI do? and AI training vs. inference.

Training and Inference
In traditional data processing, an application is a single program, although it can be made up of several modules. In AI, language models are trained to provide the data for the inference software. AI chatbots such as ChatGPT, Gemini and Claude are essentially two-part systems: a trained model and inference. In addition, chatbot users are often able to choose which model to employ because models are trained on different sets of data and are fine-tuned in various ways.

A Term with Wiggle Room?
The English word "inference" implies assumption and conjecture. Apparently, in the early days of AI, "inferring" an answer seemed a safer bet than "generating" an answer, the latter implying a degree of accuracy. Perhaps the "inference" term will offer some wiggle room in a future lawsuit! See neural network and deep learning.

Different GPUs for Disaggregated Inference

To increase performance, "disaggregated inference" executes in two stages. To analyze the question, the "context" stage (prefill stage) requires a huge amount of computation (Rubin CPX GPUs). The "decode" stage (token generation) requires fast data transfer and high-speed memory (Rubin GPUs). See Vera Rubin and long-horizon context. (Image courtesy of NVIDIA.)