AI engineering glossary

A

• A2A (Agent2Agent): an open protocol for agents to communicate with other agents, started by Google and now a Linux Foundation project. MCP page
• Abstention: the model declining to answer, asking for clarification or handing off when it lacks the information. Testing abstention
• Agent: an LLM that uses tools in a loop to complete a task. What is an agent
• Agentic RAG: retrieval where an agent decides what to search and when it has enough. Agentic RAG
• API gateway: the entry point that handles authentication, limits and routing for requests. Inference pipeline
• Approximate nearest neighbour (ANN): fast vector search that trades a little recall for speed. Vector index internals
• Attention: the transformer mechanism that weighs how much each token should influence another. How an LLM works
• Audit log: a durable record of who asked what, what the system used and what it did. Audit logs

B

• Base model: a model after pretraining only, before instruction tuning. How an LLM works
• Batch invariance: a computation giving the same result for a request regardless of batch size. Nondeterminism
• Benchmark: a fixed public test used to compare models. Benchmarks vs product evals
• BERT: an early transformer encoder model from Google, widely used for search and classification. Discrete to continuous
• BM25: a keyword ranking algorithm used in search engines. Hybrid search
• Byte pair encoding (BPE): a tokenizer that merges frequent pairs into subword tokens. Tokenizer data structures

C

• Cache hit and miss: whether a stored answer was found for a request. Semantic caching
• Canary release: sending a small share of traffic to a new version first. Rollouts
• Chain of thought: a model writing intermediate reasoning before its answer. Reasoning models
• Chunking: splitting documents into retrievable pieces. Chunking strategies
• Circuit breaker: stops calls to a failing service for a while. Reliability patterns
• Citation: a link from a claim in an answer to its source passage. Hallucination
• Context engineering: deciding what the model sees on each call. Context engineering
• Context rot: performance becoming less reliable as input length grows. Context windows
• Context window: the maximum tokens a model processes in one call. Context windows
• Continuous batching: adding and removing requests from a running batch each step. Inference optimization
• Cosine similarity: a measure of how closely two vectors point the same way. Embeddings
• Cross-encoder: a model that scores a query and passage together, used for reranking. Reranking

D

• Data flywheel: usage producing data that improves the product, which brings more usage. Data flywheels
• Decode: the token-by-token generation phase, limited by memory bandwidth. Inference optimization
• Distillation: training a small model to imitate a large one. Distillation
• DPO (Direct Preference Optimization): fine-tuning on pairs of preferred and rejected answers. Preference tuning

E

• Edge inference: running models on local devices instead of servers. Run LLMs locally
• Embedding table: the matrix that maps token IDs to vectors. Embedding tables
• Embedding: a vector that represents text so similar meanings are close. Embeddings
• Error analysis: reading traces to find and count failure modes. Find failure modes
• Eval: a test for an AI feature that checks a rule rather than one exact output. Evals
• Excessive agency: an AI system able to take more actions than it should. Agent permissions

F

• Faithfulness: every claim in an answer is supported by the retrieved context. RAG in two halves
• Fallback: an alternative path when the primary model or service fails. Fallback chain
• Few-shot prompting: including examples in the prompt. Few-shot examples
• Fine-tuning: further training a model on your data to change its behaviour. When to fine-tune
• Floating point: the number format for decimals used in model weights. Precision and memory
• Function calling: a model returning a structured request to run a tool. Function calling

G

• GGUF: a file format for quantized models used by llama.cpp and Ollama. Quantization
• Golden set: a curated eval set run on every change. CI gates
• GPU: the processor that runs model training and inference in parallel. Inference serving
• GraphRAG: retrieval using a knowledge graph built from documents. GraphRAG
• Grounding: tying an answer to provided sources. Hallucination
• Guardrail: a fast check that blocks or fixes clear failures inline. Guardrails vs evaluators

H

• Hallucination: a fluent answer that's false or unsupported. Hallucination
• HNSW: a graph-based approximate nearest neighbour index. Vector index internals
• Human in the loop: a person reviewing or approving AI outputs or actions. Human in the loop
• Hybrid fleet: several model types behind one router. Hybrid fleet
• Hybrid search: combining keyword and vector search. Hybrid search
• HyDE: searching with a hypothetical answer generated from the question. Query rewriting

I

• Inference: running a trained model to generate output. Inference pipeline
• Instruction tuning: training a base model on instructions and answers. How an LLM works
• IVF: a vector index that searches only the nearest clusters. Nearest neighbour search

J

• Jailbreak: an input that gets a model to ignore its safety rules. Prompt injection
• JSONL: one JSON record per line, the common format for LLM datasets. Data formats

K

• Knowledge graph: entities and relationships stored as nodes and edges. GraphRAG
• KV cache: stored attention keys and values for past tokens. KV cache

L

• Latency: the time from sending a request to seeing the response start. Latency and streaming
• LLM-as-judge: using a model to grade another model's output. Check your LLM judge
• Logit: a raw score per token before softmax. Logits and softmax
• LoRA: fine-tuning small low-rank adapter matrices instead of all weights. LoRA and QLoRA
• Lost in the middle: models using information in the middle of long inputs less well. Context windows
• LRU cache: a cache that evicts the least recently used entry, built from a hash map and a linked list. Hashing and caching

M

• MCP (Model Context Protocol): an open standard for connecting AI apps to tools and data. MCP
• Memory (agent): stored information an agent reloads into context. Agent memory
• Micro-batching: grouping waiting requests for a short time so the GPU runs them together. Queues and batching
• Mixture of experts: an architecture that activates only some sub-networks per token. AI system design
• Model router: a component that picks a model per request. Model routing
• MRR: mean reciprocal rank, how high the first relevant result appears. RAG evaluation
• Multi-tenancy: serving many customers from one system with isolation. Multi-tenant platforms
• Multimodal model: a model that takes images, audio or video as well as text. Multimodal models

N

• N-1 testing: replaying the first N-1 turns of a real conversation and testing turn N. Multi-turn evals
• Nondeterminism: different outputs for the same input. Nondeterminism

O

• Observability: being able to see what each request did, step by step. Observability
• Open-weight model: a model whose weights you can download and run. Model families
• Orchestration: the layer that runs an agent's steps and state. Agent frameworks

P

• PagedAttention: managing KV cache memory in pages to reduce waste. Inference optimization
• pass@k and pass^k: at least one of k tries succeeds, versus all k succeed. Evaluate agents
• PEFT: parameter-efficient fine-tuning methods such as LoRA. LoRA and QLoRA
• Perplexity (metric): how surprised a model is by text, lower means better prediction. Evaluating fine-tuned models
• PII: personally identifiable information. Data leakage and PII
• PILOT: Problem, Insight, Learning, Outcome, Traction, a problem-first way to learn and build. Learn by building
• Prefill: the parallel processing of the prompt before generation. Inference optimization
• Pretraining: training on large text data to predict the next token. How an LLM works
• Prompt caching: reusing processing of an identical prompt prefix. Prompt caching
• Prompt injection: text that makes a model follow an attacker's instructions. Prompt injection

Q

• QLoRA: LoRA on a 4-bit quantized base model. LoRA and QLoRA
• Quantization: storing weights with fewer bits. Quantization
• Query rewriting: turning a user message into a better search query. Query rewriting

R

• RAG: retrieval-augmented generation, answering from retrieved sources. RAG architecture
• ReAct: an agent pattern that alternates reasoning and tool actions. Planning patterns
• Reasoning model: a model that generates thinking tokens before answering. Reasoning models
• Recall@k: share of relevant items found in the top k results. RAG evaluation
• Reciprocal rank fusion: merging ranked lists by their ranks. Hybrid search
• Red teaming: attacking your own system to find failures. Red teaming
• Reflection: an agent critiquing and revising its own output. Planning patterns
• Reranking: reordering retrieved candidates with a more accurate model. Reranking

S

• Sampling: how the next token is picked from the probability distribution. Sampling
• Sandbox: an isolated environment for running code or tools. Tool registry and sandbox
• Semantic cache: returning stored answers for questions with the same meaning. Semantic caching
• Semantic search: search by meaning rather than exact words. Embeddings
• Shadow AI: employees using unapproved AI tools with company data. AI governance
• Small language model (SLM): a compact model for narrow, fast, cheap tasks. Model families
• Softmax: turns scores into probabilities that sum to 1. Logits and softmax
• Span: one unit of work inside a trace. What a trace records
• Speculative decoding: a small model drafts tokens that a large model verifies. Inference optimization
• Streaming: sending tokens to the user as they're generated. Latency and streaming
• Stride: how many elements to skip in flat memory to move one step along a tensor dimension. Tensors
• Structured outputs: generation constrained to a JSON schema. Structured outputs
• Synthetic data: data generated by a model for testing or training. Synthetic data
• System prompt: the instructions that set an app's rules for the model. System prompts

T

• Temperature: a setting that controls randomness in sampling. Sampling
• Tensor: an array with any number of dimensions. Tensors
• Time to first token (TTFT): how long until the first output token arrives. Latency and streaming
• Token: the unit of text a model reads and writes. Tokens
• Tool: a function an agent can call. Tool design
• Top-k and top-p: limits on which tokens can be sampled. Sampling
• Trace: the full record of one request through the system. What a trace records
• Transformer: the neural network architecture behind modern LLMs. How an LLM works
• Transition failure matrix: a grid of last good step vs first failed step. Transition failure matrix
• Trie: a tree that stores strings by shared prefixes, used in tokenizers and prefix caches. Tokenizer data structures

V

• Vector database: a store for embeddings with fast similarity search. Vector databases
• Vector: a list of numbers. Discrete to continuous
• vLLM: an open-source high-throughput LLM serving engine. vLLM vs SGLang vs Ollama

W

• Workflow (LLM): a sequence of model calls and tools defined by your code. Workflows vs agents

Z

• Zero-shot prompting: asking a model to do a task with no examples. Few-shot examples

From the floor

I explain a new term the same way in every session: what problem it solves, what happens when you use it, and what you do when it doesn't work, because a definition you can't connect to a component in your own build doesn't stay with you.

Related

• Start here
• Comparisons
• Roadmap
• LLM fundamentals

Sources

• OWASP Top 10 for LLM Applications 2025