How much math do you need for AI engineering?
You don't need to finish a math course before you start AI engineering, and you don't need to be a math genius. You need the math behind the components you build, learned when you build them: vectors and cosine similarity when you build retrieval, probability when you work with sampling and evals, matrix multiplication when you study attention, and derivatives when you fine-tune. If you want to become an ML engineer or implement research papers, you will need to go much deeper.
Why this question stops so many people
Almost everyone who starts AI engineering hears the same thing at some point: you need to be a math genius, maybe with a PhD. So people open a linear algebra course, then a calculus course, then a statistics course, and months later they still haven't built anything with AI. Many give up before they reach the part they were excited about.
That advice is a myth for most AI engineering work. Libraries like PyTorch, TensorFlow and LangChain, and every model API, have already done the heavy lifting. The calculus, linear algebra, statistics and probability are packaged inside functions, classes, modules and packages. Your job is not to derive the math by hand, it is to know which tool to use and why, which means understanding the curves, not calculating them.
Not up front, but not never
That doesn't mean you can ignore math forever. To implement artificial intelligence, the only tool we really have is mathematics, and every model you call is matrices, vectors and probabilities underneath. If you know the math, you understand what is going on behind the scenes, you debug faster, and you make better decisions about which method to use.
So my answer has two parts. Math is not mandatory to start, and if you already know it, that is a plus. But when you go inside a component, you learn the math that component uses, because that is when the math makes sense and when it sticks.
Why learning math first doesn't work well
The old way to learn machine learning was to study the whole of mathematics first and then move to algorithms. That way is slow, and most of us already studied a lot of this math in school and college without knowing where it would be used.
The better way is the other way around. First learn the algorithm or the component, see what it does and why you need it, and then learn the math behind that algorithm. If a problem uses logistic regression, learn how logistic regression works and then the math it uses, not every topic in a statistics textbook. When you see the math as the reason a component works, you build intuition, and intuition is what you will use in real projects and interviews.
Four areas of math used in AI
Most of the math in machine learning and AI comes from four areas. Linear algebra and statistics are used the most, and you will use some probability and some calculus too.
Linear algebra
Linear algebra is the math of vectors and matrices, and in AI almost everything becomes a vector or a matrix.
A colour image, for example, is three two-dimensional matrices, one each for red, green and blue, where every number is the intensity of one pixel. When you flip an image left to right, you reverse the order of the columns in each matrix. When you transpose a matrix, you swap its rows and columns, which mirrors the image across its diagonal, and if you transpose and then flip, you rotate the image by 90 degrees. These are the same operations that image libraries and vision models run internally.
Text works the same way once it is converted to numbers. Every token becomes a vector called an embedding, and words with similar meanings end up close to each other in that vector space. Each layer of a neural network, including a transformer, is mostly matrix multiplication: input vectors multiplied by learned weight matrices.
What you need as an AI engineer:
- Vectors and what the dimensions of a vector mean.
- Dot product, magnitude and cosine similarity, because they decide how retrieval finds relevant text.
- Matrix multiplication and shapes, because most errors you will hit in PyTorch or NumPy are shape mismatches.
- An intuition for what it means to reduce dimensions or approximate a large matrix with smaller ones, which is the idea behind methods like LoRA.
Statistics
Statistics gives you two kinds of tools for working with data.
Descriptive statistics describes the data so you can understand it: mean, median, mode, frequencies, spread and distributions. You can describe one variable at a time (univariate, like the height of everyone in a class), the relationship between two variables (bivariate, like whether having a master's degree changes salary), or many variables together (multivariate, like a whole table of customer data).
Inferential statistics goes further and helps you draw conclusions or make predictions from data, with methods like hypothesis tests and regression.
As an AI engineer, statistics shows up most in evaluation. When you say your new prompt is better, you are making a statistical claim, so you need to know whether 50 test cases are enough, whether the difference between 82% and 85% is real or noise, and how to compare two versions fairly.
Probability
A language model doesn't pick the next word, it produces a probability for every token in its vocabulary and then samples one. The softmax function turns the model's raw scores into those probabilities. Temperature changes how spread out the probabilities are, which is why a higher temperature gives more varied answers, and top-p sampling limits the choice to the most likely tokens.
What you need as an AI engineer:
- Probability distributions and what sampling from one means.
- Softmax and why it makes all probabilities add up to 1.
- Conditional probability, because the model predicts the next token given everything before it.
- Enough to understand why the same prompt can give different answers, and how to make outputs more consistent.
Calculus
Calculus is how models learn. During training, the model makes a prediction, a loss function measures how wrong it was, and derivatives tell you in which direction to change each weight to reduce that loss. Repeating that step many times is gradient descent.
As an AI engineer, you rarely compute derivatives yourself, because frameworks do it automatically. But when you fine-tune a model, you set a learning rate, watch a loss curve and decide when to stop, and all of that is easier when you understand what the gradient is doing.
What you need as an AI engineer:
- What a derivative and a gradient mean, as a direction and a rate of change.
- What a loss function is.
- How gradient descent and the learning rate work, at the level of intuition.
Where each math topic shows up in your builds
This table maps the components you will build to the math they use, so you know what to learn when you reach each one.
| When you build or work with | You will use | Learn it from |
|---|---|---|
| Embeddings and semantic search | Vectors, dot product, magnitude, cosine similarity | Embeddings |
| A vector database | Distance metrics, nearest neighbour search | Vector databases |
| Sampling and output settings | Probability distributions, softmax, temperature | Logits and softmax |
| Attention and transformers | Matrix multiplication, softmax, scaling | LLM fundamentals |
| Tensors in PyTorch | Shapes, broadcasting, matrix operations | Tensors |
| Fine-tuning and LoRA | Gradients, loss, learning rate, low-rank matrices | When to fine-tune |
| Quantization | Number formats, precision, rounding error | Floating point precision |
| Evals and A/B tests | Sampling, confidence, comparing two versions | Evals and reliability |
| Images and vision models | Matrices, transpose, convolution | AI data structures and algorithms |
Worked example: the math inside a vector store
The best way to see how little math you need to start, and why it still matters, is to build one component from scratch. In one of our sessions, we built a small vector store in plain Python before using any library.
The idea is simple. Each sentence is converted into a vector by an embedding model. Sentences with similar meaning point in similar directions, so "cat", "kitten" and "sofa" end up close to each other, and "stock market" ends up far from all of them. To measure how close two vectors are, we use cosine similarity, which compares their directions.
Cosine similarity needs three small functions. The dot product multiplies the two vectors position by position and adds the results. The magnitude is the length of a vector, the square root of the sum of its squared values. And cosine similarity is the dot product divided by the product of the two magnitudes, which gives a number between -1 and 1, where 1 means the same direction.
import math
def dot_product(a, b):
return sum(x * y for x, y in zip(a, b))
def magnitude(v):
return math.sqrt(sum(x * x for x in v))
def cosine_similarity(a, b):
return dot_product(a, b) / (magnitude(a) * magnitude(b))
store = {
"the cat sat on the mat": [0.9, 0.1, 0.0],
"a kitten on the sofa": [0.8, 0.2, 0.1],
"the stock market fell today": [0.0, 0.1, 0.9],
}
query = [0.85, 0.15, 0.05]
results = sorted(
store.items(),
key=lambda item: cosine_similarity(query, item[1]),
reverse=True,
)
for text, vector in results:
print(round(cosine_similarity(query, vector), 3), text)
The vectors here are tiny and written by hand so you can follow the numbers. Real embedding models return vectors with hundreds or thousands of dimensions, for example the all-MiniLM-L6-v2 model on Hugging Face returns 384, but the math is exactly the same.
After you build this, three things that looked like magic become clear. You know why a vector database is fast or slow, because comparing a query with every stored vector gets expensive as the store grows, which is why real databases use approximate nearest neighbour indexes. You know why the choice of embedding model matters, because it decides where every sentence lands. And you know why retrieval sometimes returns a chunk that shares words with your question but not its meaning.
How deep you need to go for each role
The amount of math depends on the work you want to do.
As an AI engineer building products on top of models, you need the intuition and the basic operations from each section above, learned as your builds need them.
As an ML engineer, you need more, because you train and tune models and sometimes customise them. If you want to implement a method from a research paper, you need to read its equations and turn them into code, and that needs a solid base in linear algebra, calculus, probability and statistics.
As a researcher creating new methods, math becomes the main language of your work, and a deep, formal background is expected.
If you are not sure which of these you want, start as an AI engineer, and go deeper when your work or your curiosity pulls you there. That is how I learned it too, because I was interested in mathematics, so I studied it on its own, but I would not ask everyone to do that before they build their first system.
A math plan that follows your builds
Here is the order I recommend, matched to the roadmap.
- Before your first LLM call: nothing beyond school arithmetic.
- When you study tokens and embeddings: vectors, dot product, magnitude, cosine similarity.
- When you build retrieval: distance metrics and nearest neighbour search.
- When you set temperature and work with model outputs: probability distributions and softmax.
- When you build evals: descriptive statistics, sampling and how to compare two versions.
- When you study attention: matrix multiplication and shapes.
- When you fine-tune: derivatives, gradients, loss and learning rate.
- When you want to implement papers: go back and study each area formally.
Resources when you want to go deeper
- Essence of Linear Algebra by 3Blue1Brown, a free video series that builds visual intuition for vectors and matrices.
- Mathematics for Machine Learning by Deisenroth, Faisal and Ong, a free book that covers all four areas with machine learning in mind.
- Khan Academy's statistics and probability course, for the basics of descriptive and inferential statistics.
From the floor
In a linear algebra session, one builder asked a question that many beginners have: we understand transpose, cross product and concatenation, but how do these help me build, for example, an app that reads someone's facial expression and predicts their mood? My answer was that most of these operations are taken care of by the algorithm itself, and you are building the intuition of how they are used inside it. If you build a model from scratch, you will write these operations directly, converting images into matrices and applying them, and if you use a library, you need the intuition to understand what the library is doing when something goes wrong.
When I start the machine learning part of a cohort, I cover only the fundamentals of statistics and linear algebra in the first sessions, and the more complex concepts come later, alongside the algorithms that use them. That is the order that helps builders understand faster.
Related
- Embeddings
- Vector databases
- AI data structures and algorithms
- AI engineer vs ML engineer vs data scientist vs software engineer