AI engineering FAQ: questions builders asked in the sessions
These are questions builders and leaders asked me in live sessions, AI Engineer HQ cohorts, Build30 and workshops, with the answer I gave, cleaned up, and checked against current sources where the answer depends on a fact that can change. Each answer starts with the direct answer, then explains it, and links to the docs page that goes deeper. Where my answer in the session was incomplete or wrong, the answer here says what the correct version is.
How to use this FAQ
| If you are | Start with |
|---|---|
| Moving into AI engineering | Career and learning |
| Confused about how LLMs behave | LLM fundamentals |
| Calling LLM APIs for the first time | APIs, tools and setup |
| Building RAG | RAG and embeddings |
| Building agents | Agents |
| Running or tuning your own models | Fine-tuning, quantization and local models |
| Designing for production | Production and system design |
| Leading AI work | Leaders and business |
Career and learning
Q: Are an ML engineer and an AI engineer the same role?
They overlap, and companies use the titles loosely, so read the job description, not the title. An ML engineer is usually expected to know ML algorithms, how models are trained and how those systems are deployed. An AI engineer is usually expected to build products on top of models: LLM APIs, retrieval, agents, evals and deployment. In both cases, companies hiring for a new role expect you to know the newer technologies that role uses.
Asked in a Python prep session for AI Engineer HQ. Read more: AI engineer vs ML engineer
Q: Should I start with a harder project, or go back and finish the simpler ones first?
You don't need to go back to the simpler projects. Work on the harder project, and when it uses topics you don't know yet (like QLoRA in a fine-tuning project), ask your coding assistant to explain them as you go, and at the end ask it to write a manual of every topic the project used, module by module, so you can study what you built.
Asked during the Model Smith fine-tuning build. Read more: Learn by building
Q: Do I need Claude Code, Copilot and Codex, or is one coding assistant enough?
One is enough. Jumping between tools costs you time, and one good coding agent can do what the project needs. You can still use Copilot alongside Claude Code on an existing project if your team already uses it.
Asked during the hallucination citation verifier build. Read more: How coding agents are built, and how to work with them
Q: Do AI engineering interviews focus on algorithms?
It depends on the interview. Engineering interviews can include algorithm and technical questions, and AI system design rounds ask you to design and explain a system out loud, so prepare for both.
Asked during the Agent Forge build. Read more: AI system design interview and data structures and algorithms for AI engineers
Q: Do certifications help me get an AI engineering job?
Mostly no, with cloud certifications as the exception for engineering roles. Before you spend time and money on a certification, check whether companies hiring for your role value it, and what you'll learn from it. Deployed projects you can explain do more for you than a certificate.
Answered in the career webinar and the portfolio session. Read more: AI engineer portfolio
Q: Should I go for a job or freelancing first?
Pick one and focus on it. If you're early in your career, get a job first to understand how the industry works, and freelance on the side later. My own path was different, I started freelancing in my second year of college and never took a job, because I knew I wanted to start a company, so treat that as one path, not a template.
Asked in the career webinar. Read more: Freelancing and consulting
Q: How do I find freelance clients?
Clients give the project to someone who has already solved their problem. If you've solved it before and written about it in a blog or a portfolio, the client is much more likely to reach out to you. Join communities and share useful work consistently, instead of posting that you're available for work.
Asked in the career webinar. Read more: Freelancing and consulting
Q: Do I need many followers before LinkedIn shows my posts to more people?
No. More followers help, but what spreads a post is engagement from real people, people commenting and discussing the idea. LinkedIn first shows your post to some of your followers, and if they engage with it, it shows the post to a wider audience, even if you only have a few thousand followers.
Asked in the LinkedIn content session. Read more: AI engineer portfolio
LLM fundamentals
Q: What exactly is a large language model?
It's a model that predicts the next token. A token is a word or a piece of a word, so "token" might be split into "to" and "ken". You give it text, it predicts the next token, adds it to the text and repeats, which is how it writes a full answer.
Asked in the ReAct agent orientation session. Read more: How an LLM works
Q: Does ChatGPT remember everything I've told it?
The model itself doesn't. The model starts fresh with every request and doesn't update its weights from your chats. What feels like memory is the app storing your chat history or saved memories and putting the relevant parts back into the context, so it's limited by the context window and by what the app chooses to include.
Asked in the AI ecosystem session. Read more: Context windows and agent memory
Q: Why does an LLM give a different answer when I ask the same question again?
Because the next token is picked from a probability distribution, and with sampling, a different token can be picked each time. Even at temperature 0, outputs can differ because of how the server batches requests, which changes the order of floating point operations on the GPU.
Asked in the session on nondeterminism. Read more: Why LLMs give different answers at temperature 0
Q: Is a different answer to the same question a hallucination?
No, those are two different problems. Nondeterminism means the wording or approach changes between runs. Hallucination means the answer contains wrong or unsupported information, and it can happen in every run.
Asked in the session on nondeterminism. Read more: LLM hallucination
Q: Can we tell the GPU to process things in a fixed order so the output is always the same?
Yes, it's possible, with a cost. In the session I pointed to the Thinking Machines post on this, and the main cause it describes is that kernels aren't batch invariant, so the same request gives slightly different numbers depending on how many other requests are in the batch. Their fix is batch-invariant kernels, which make outputs reproducible but run slower, so most providers don't use them by default.
Asked in the session on nondeterminism. Read more: Why LLMs give different answers at temperature 0
Q: Why can't the LLM calculate this? Isn't it simple math?
It might calculate it correctly, but you can't rely on it. An LLM predicts the answer as text from patterns in its training data, so it can produce a number that looks right and isn't. In the session the model got the answer right, which is the point: it's sometimes right, and for exact values you give the agent a calculator tool or compute in code.
Asked in the agentic workflow session. Read more: Exact values pattern
Q: Do I need to fine-tune a model to do grammar correction or summarisation?
Usually not. A base model trained only on general data needs training for specific tasks, but the chat models you use through APIs have already been instruction tuned, so they can correct grammar or summarise from a prompt. Fine-tune when prompting and examples don't get the behaviour you need.
Asked in the AI overview session. Read more: When to fine-tune
Q: How many transformer blocks are there in a model?
Many, and the number depends on the model. Each block applies attention and then a feed-forward network, and passes its output to the next block, which repeats the same steps. Small models have a few dozen blocks, and the largest have around a hundred or more.
Asked in the transformer architecture session. Read more: How an LLM works
Q: What's the difference between an encoder and a decoder?
An encoder processes the input and creates a contextual representation of it. A decoder generates output, one token at a time. BERT-style models are encoders and are used for search and classification, and GPT-style LLMs are decoders.
Asked in the transformer architecture session. Read more: How an LLM works
Q: At the first embedding step, does a token know anything about the sentence around it?
No. The embedding table gives each token the same vector wherever it appears, so "apple" the fruit and "Apple" the company start the same. Attention in the transformer layers then mixes in context from the other tokens.
Asked in the Build30 tokenization session. Read more: Embedding tables
RAG and embeddings
Q: How is RAG different from an LLM?
An LLM is a model, and RAG is a technique you use with it. RAG retrieves relevant information from your own data at query time and puts it in the prompt, so the model can answer about things it wasn't trained on, like this quarter's numbers or your company's policies.
Asked in the RAG session. Read more: RAG architecture end to end
Q: Is a query vector the same as an embedding vector?
Yes. It's an embedding of the user's query, and we call it a query vector only because it's the query being processed.
Asked in the LLM inference pipeline lecture. Read more: Embeddings
Q: Does every file, Excel, CSV, PDF or Word, have to become vectors?
For semantic search, yes, the text in them is converted into embeddings, because the model works with numbers, not English. Exact values, like order numbers or amounts in a spreadsheet, are often better kept in a database and queried directly, alongside the vectors.
Asked in the AI system design session. Read more: Discrete to continuous values and exact values pattern
Q: What does a negative number in an embedding mean?
On its own, nothing you can read. Each number is one dimension of a learned space, and the sign only says which side of that dimension the word sits on. What carries meaning is the whole vector compared with other vectors, for example by cosine similarity.
Asked in the AI data structures lecture. Read more: Embeddings
Q: What is cosine similarity?
It measures how closely two vectors point in the same direction, from -1 to 1. Words and sentences used in similar ways get vectors that point in similar directions, so "king" and "queen", or "charger" and "battery", score high.
Asked in the Build30 RAG pipeline session. Read more: Embeddings
Q: Is ChromaDB a vector database?
Yes. ChromaDB is an open-source vector database, which is why we use it for local projects.
Asked during the Insight Lens build. Read more: Vector databases and pgvector vs Qdrant vs Pinecone
Q: What chunking strategy should I use for a small dataset?
With small data and short paragraphs, one or two sentences per chunk works. For larger documents, chunk by structure (sections and headings), and choose the size by testing retrieval on your own questions, not by picking a number.
Asked in the Build30 RAG pipeline session. Read more: Chunking strategies and choose chunk size with evals
Q: My support content is HTML pages, not PDFs. Can I still use it?
Yes. HTML is text, so you can extract it with a few lines of Python by removing the tags, then chunk and embed it like any other text.
Asked in the customer support agent project. Read more: Document parsing
Q: Can I deploy the customer support agent on my company's own data?
Yes. Replace the sample data in the code with your company's documents, and then add what production needs: access control, evaluation on your real questions, and deployment.
Asked in the customer support agent project. Read more: Customer support assistant reference
Agents
Q: If the LLM can calculate, where does the agent come in?
The agent decides to use a tool, like a calculator, and decides what to pass to it. For a question like "what's 15% of this amount?", the agent has to turn the sentence into an expression, the amount multiplied by 0.15, and send that to the calculator, because you can't pass a whole sentence to a calculator, and that decision is the intelligent part.
Asked in the agentic workflow and agent tool use sessions. Read more: What is an AI agent and tool design
Q: Is an agent just a workflow?
It can run one, but it's not the same thing. A workflow is a fixed sequence your code defines, like booking a flight and then the hotel. An agent decides the steps itself, which tools to call and in what order, and that's useful only when the steps can't be fixed in advance.
Asked in the agentic workflow session. Read more: Workflows vs agents
Q: What do "tools" and "thinking" mean for an agent?
Tools are external functions the agent can call, an API, a calculator, a search. Thinking is the model reasoning about why and how it's doing a task before it acts.
Asked in the customer support agent project. Read more: Planning patterns and reasoning models
Q: Is LangChain only for LLM apps, so I need LangGraph for agents?
You can build agents with LangChain alone, but LangGraph gives you more control. LangChain was built mainly for integrating LLMs into applications, and LangGraph was built separately for agents, with state, loops and checkpoints.
Asked during the DevFix auto agent build. Read more: LangGraph vs CrewAI vs Microsoft Agent Framework
Q: Why did we install Postgres for the agent?
To store the agent's information, its state and logs, so a run can be inspected and resumed instead of living only in memory.
Asked during the Agent Forge build. Read more: Agent state, checkpoints and recovery
Q: Does a skill file I write for Claude Code also work with Copilot or other assistants?
Increasingly, yes. Anthropic published Agent Skills as an open standard in December 2025, and other coding agents have adopted the same SKILL.md format, but check that the assistant you use supports it before relying on it.
Asked during the DevFix auto agent build. Read more: How coding agents are built
Q: Is n8n free to host?
Yes, you can self-host the Community Edition for free. In the session I said n8n was proprietary with a short free trial, and that was wrong: n8n is source-available under its own Sustainable Use License, the self-hosted Community Edition is free, and the 14-day free trial is for n8n Cloud. Check the license terms before you build a commercial product on it.
Asked in the embeddings and vector databases session. Read more: Agent frameworks
APIs, tools and setup
Q: What is the API key for?
Authentication. In the chat interface you're logged into your account, but when your code calls the model there's no login screen, so the API key identifies your account and is used for access and billing. Keep it out of your code and repository.
Asked in the agent tool use session. Read more: Secrets management
Q: Does an OpenAI API key work with other providers' models?
No. Each provider gives you its own key and its own SDK, so a Google model needs a Google key. Frameworks like LangChain, or a gateway, let you switch providers with small code changes, but you still need each provider's key.
Asked in the customer support agent build. Read more: Working with LLM APIs and model routing and gateways
Q: Can my Python agent call an existing API written in .NET or Java?
Yes, directly, with no special library. APIs communicate in JSON over HTTP, so the language the API was written in doesn't matter to the agent calling it.
Asked in the customer support agent build. Read more: Tool design
Q: If I use LangChain, do I still need TensorFlow?
No. LangChain calls models through APIs, and libraries like it are built on top of other libraries, so you install what they need, not a deep learning framework.
Asked in the customer support agent build. Read more: Python for AI engineers
Q: Do I still need a cloud provider if I build my agent with LangChain or Claude Code?
Yes. LangChain is a library you build with, and Claude Code is a tool you write code with, while a cloud provider is where the application runs so users can reach it.
Asked in the customer support agent build. Read more: Cloud basics: your first deploy
Q: Where can I get a free API key to practise with?
Google AI Studio, Groq and OpenRouter all have free tiers, with rate limits. Free tiers can use your prompts to improve their products (Google says so in its terms for free use outside the EU, UK and EEA), so don't send real company or personal data through them, and check the current limits, because they change often.
Asked in the RAG agent workshop. Read more: Working with LLM APIs
Q: Do I have to buy Azure Machine Learning to use it?
No. It's pay as you go, so you're charged for what you use, with no upfront cost.
Asked in the fine-tuning evaluation session. Read more: Cloud basics
Q: What is a pull request for?
To check code before it's merged. Without one, a bug you push goes straight to every developer on the project, while a pull request lets someone review it first, usually the maintainer or a senior developer.
Asked in the LinkedIn AI comment assistant build. Read more: Git and GitHub for AI projects
Q: I have Python 3.10, 3.11 and 3.13 installed. Which should I keep?
You can keep them, as long as each project's virtual environment uses the version it needs. In the financial RAG project the dependencies worked with 3.11, so we created the environment with 3.11.
Asked in the RAG project lab. Read more: Python for AI engineers
Q: Should I add comments to my code?
That's your choice. I don't add comments to my own code, which is why the prompts I share say so, but if you're learning, keep them.
Asked during the Agent Forge build.
Q: Should I clear or compact the context in Claude Code between sessions?
Usually clear. Clear starts a new session with empty context, and compact summarises the conversation so far to free up space. I generally clear, because the project's CLAUDE.md and architecture files hold what the next session needs, so I tell it to read those and continue.
Asked during the citation verifier build and the AI data structures lecture. Read more: How coding agents are built and context compression
Fine-tuning, quantization and local models
Q: Why fine-tune if we can already use RAG?
They solve different problems. RAG brings in external data at query time, like this year's earnings, and fine-tuning trains the model further on a specific dataset so it behaves differently, like classifying movie reviews or understanding legal terms.
Asked in the fine-tuning fundamentals session. Read more: RAG vs fine-tuning
Q: If I download and run a model locally, does my data still go to a public model?
No. The weights are on your machine, so the question and the answer both stay there, while with a hosted chat app your request goes to the provider's servers.
Asked in the edge inference session. Read more: Run LLMs locally
Q: How much accuracy do I lose when I quantize a model?
It depends on how far you go. My rough experience is that 8-bit loses very little and 2 to 3 bits can lose a lot, and research says the same: Jin and others (2024) found 4-bit models retain performance comparable to the originals, with a notable drop at 3 bits or lower. Test the quantized model on your own eval set before deciding.
Asked in the edge inference session. Read more: Quantization and floating point precision
Q: How do I check the quality of a quantized model's output?
Compare it with the full-precision model on the same eval set built from your own task, and look at the failures, not just the average. In the session I said this needed a session of its own, and the evals pages cover it.
Asked during the Insight Lens build. Read more: Evaluating fine-tuned models
Q: Does all quantization happen through llama.cpp?
No, but a lot of local tools use it. Ollama and LM Studio run models through llama.cpp under the hood, while other methods, like the bitsandbytes 4-bit used in QLoRA, work differently.
Asked in the edge inference session. Read more: Quantization and vLLM vs SGLang vs Ollama
Q: What is Unsloth?
An open-source library for faster, more memory-efficient fine-tuning, which also publishes quantized versions of popular models. In the session a builder noticed its models ran faster, and that comes from its optimisations and quantization.
Asked during the Insight Lens build. Read more: LoRA and QLoRA
Q: Did Bloomberg use Ollama for BloombergGPT?
No. Ollama is a tool for running models locally, not where models are trained. BloombergGPT was trained from scratch by Bloomberg on a mix of its own financial data and general data, which is a bigger effort than the fine-tuning I described in the session.
Asked in the customer support agent project. Read more: Cloud APIs vs open-source LLMs
Q: Do I need a GPU, or can I use Google Colab?
You need a GPU to run models fast, and Colab gives you one, with paid options for better GPUs and longer use. For learning, small models (1 billion parameters or so, quantized) run on a normal laptop.
Asked in the session on nondeterminism. Read more: Run LLMs locally
Production and system design
Q: Why put an API gateway in front of the LLM?
To have one entry point for every model call: authentication, rate limits, caching, routing between models and logging. The gateway can also call an embedding model to check a semantic cache before the request goes further.
Asked in the LLM inference pipeline lecture. Read more: LLM inference pipeline and model routing and gateways
Q: Does a request always go to the cache first, then the vector database?
Yes, in that design. If you remove the cache, every repeated question goes through the whole pipeline again, which costs time and money.
Asked in the LLM inference pipeline lecture. Read more: Semantic caching
Q: Is the cache prompt caching or semantic caching?
In the gateway, it's semantic caching: it matches the meaning of the user's query against past queries. Prompt caching is different, the provider reuses the processing of an identical prompt prefix to cut cost and latency.
Asked in the LLM inference pipeline lecture. Read more: Semantic caching and prompt caching
Q: How does the router decide whether a query is simple or complex?
Start with rules (if this kind of question, go here), and if that's not enough, fine-tune a small model or use embeddings of labelled examples to classify queries, which is semantic routing. A simple query is a straightforward answer like "what is the capital of India", and a complex one needs several steps or tools.
Asked in the LLM inference pipeline lecture and the cloud APIs vs open-source session. Read more: Design a model router
Q: Doesn't adding an intent router increase latency?
Yes, it does. So you test different solutions and keep the one where the router's cost in time is smaller than what it saves, for example rules or a small classifier instead of another LLM call.
Asked in the LLM inference pipeline lecture. Read more: Reduce latency for a chat assistant
Q: How do you manage session memory for 10,000 users who come back a week later?
Keep active sessions in a fast cache with a buffer of recent turns, write summaries and long-term facts to durable storage when sessions go idle, and load only what's relevant when the user returns, keyed by user so context never bleeds between users. In the session I said I didn't have a complete answer ready, so the full answer is a page of its own.
Asked in the LLM inference pipeline lecture. Read more: Keep session memory for 10,000 returning users
Q: Should we go cloud, on-prem, or both?
Often both. A hybrid fleet sends routine requests to a cloud API and sensitive or specialised ones to a model you host, and either way you count the hidden costs: for open source, the engineers to integrate, deploy and monitor it, and for APIs, token bills that grow with usage.
Asked in the cloud APIs vs open-source session. Read more: Cloud APIs vs open-source LLMs and hybrid fleet pattern
Leaders and business
Q: Why would a company buy a private knowledge engine when Copilot can search its files?
Because some companies won't send their data to a cloud provider at all. Companies hesitant about external LLMs don't want their data on another company's servers, and in defence and law enforcement the systems often have no internet access, so the solution has to run on their own servers.
Asked in the AI builder architect manual session. Read more: Enterprise knowledge engine reference and RAG for regulated data
Q: How should we weight criteria when scoring AI use cases?
Weight business value highest, because return is what matters most, then data readiness and similar factors, then strategic alignment. In the scoring sheet we used, business value was multiplied by 3, data readiness by 2 and strategic alignment by 1, and a participant pointed out that without weights everything sits on the same scale and the score stops being useful. For regulated industries, risk may deserve the highest weight.
Asked in the AI leadership cohort. Read more: How to pick your first three AI use cases
Q: Where do the value metrics for a use case come from?
From the company, not from the AI team. The business decides which numbers matter, like how long an underwriting decision takes today and whether it could happen in a day, and the AI work is measured against those numbers.
Asked in the AI leadership cohort. Read more: Estimating AI ROI and cost
Q: How do I take payments in a product I build?
Use a payment provider like Stripe: you add its API keys to your code, it handles the payment page, and it sends your application the payment details as JSON when someone pays. The API keys identify your Stripe account, which is why the integration needs them.
Asked in the AI builder architect manual session. Read more: Secrets management
From the floor
Every question on this page came from a live session, a builder or leader asking while we were building or discussing something, and I've kept the answer I gave unless it was incomplete or wrong. Where it was wrong, like my answer about n8n's license, the answer says so and gives the correct version.
Related
Sources
- Defeating Nondeterminism in LLM Inference, Thinking Machines Lab
- A Comprehensive Evaluation of Quantization Strategies for Large Language Models, Jin and others (2024)
- BloombergGPT: A Large Language Model for Finance, Wu and others (2023)
- Choose your n8n, n8n documentation
- Agent Skills: Anthropic's next bid to define AI standards, The New Stack
- Free LLM APIs compared, OpenRouter (2026)
- Unsloth