Cloud basics for AI engineers: your first deploy
Deploying means moving your application from your laptop to a server where other people can use it. For your first AI deploys, use a managed platform that builds your Docker image or your repository and runs it for you, like Render, Railway, Google Cloud Run or AWS App Runner, and move to virtual machines or Kubernetes only when you need the control. Whatever you choose, set your secrets as environment variables, add a health check, read your logs, put a limit on spending, and test the deployed version with real requests.
What deployment means
When you build a project, it runs on your machine, in your environment, for you. Deploying means putting it somewhere other people can reach it, on a server in the cloud or in a company's own data centre, the same way you reach any website or product.
Think about ChatGPT. Millions of people send it requests, so the model isn't running on someone's laptop, it is deployed on servers, and when you type a question, the website calls an API, the API runs code that uses the model, and the answer comes back to your screen. Your project goes through the same path at a much smaller scale.
Why deployment teaches you so much
Deployment is where you find out what your project can't handle.
On your machine, you are the only user, and you use it the way you designed it. Once it is live, real users do things you never expected. They paste a 50 page document into a field built for a question, send an empty request, click the button 10 times, or type 2.5 into a quantity field. On an e-commerce site, the quantity is a dropdown, not a text box, for exactly that reason.
Every company you join has a product or a service running on a server somewhere, so they expect you to understand deployment, and a deployed project with a link is much stronger evidence than a notebook.
Phases of getting to production
In our sessions, I break the path to production into six phases:
- Development and local testing, in VS Code or your editor, including the API and its tests.
- Containerization with Docker.
- Cloud setup: choosing a provider and preparing what the app needs.
- Deployment: pushing the container or code to that provider.
- A frontend, if the users need one.
- Monitoring, security and maintenance.
This page covers phases 3 and 4, and the basics of 6.
Where you can deploy
There are four main kinds of targets, and they trade ease of use for control.
Managed platforms
You connect your GitHub repository or give the platform a Docker image, and it builds, runs, restarts and gives you a URL with HTTPS. Render, Railway, Fly.io, Google Cloud Run, AWS App Runner and Azure Container Apps work this way. Hugging Face Spaces is a good option for demos of models, and Vercel and Netlify are built for frontends.
This is where I recommend starting. You don't manage servers, operating system updates or networking, so you can focus on the application, and most of these platforms have a free tier or a low starting price.
The free tiers have limits you need to know. On Render, for example, a free web service spins down after 15 minutes without traffic, and the next request has to wait about a minute while it starts again, which is fine for a portfolio demo and not fine for a product.
Virtual machines
A virtual machine (VM) is a server you rent, like an AWS EC2 instance, a Google Compute Engine VM, an Azure VM or a DigitalOcean droplet. You connect to it with SSH, install what you need, and run your app, usually with Docker.
You get full control: any library, any port, a GPU if you pay for one, and the ability to run several services on one machine. In exchange, you are responsible for everything: security updates, firewall rules, restarts after crashes, HTTPS certificates and backups.
Serverless functions
AWS Lambda, Google Cloud Functions and Azure Functions run a function only when a request arrives, and you pay only for the time it runs. They work well for small, short tasks, like a webhook that classifies an incoming email. They are a poor fit for long LLM calls, streaming responses, large dependencies like PyTorch, and anything that needs to keep a model loaded in memory, because of time limits, package size limits and cold starts.
Kubernetes
Kubernetes runs and scales many containers across many machines. It is what larger teams use for complex systems, but it adds a lot of operational work. You don't need it for your first projects, and many companies run AI products in production without it.
Cloud or on-premises?
Some companies can't send their data to a public cloud at all. Defence, some government work, some healthcare and financial data, and other sensitive domains often require on-premises deployment, on servers the company owns, inside its own network.
On-premises gives full control over data and can be cheaper when usage is high and steady, but it needs hardware bought up front, an IT team to run it, and scaling is limited by the hardware you have. The cloud lets you start small, scale up and down, and leave maintenance to the provider, which is why it fits most new projects, and in between, a hybrid setup keeps the sensitive data on-premises and runs the rest in the cloud.
Your first deploy, step by step
This is the flow for a FastAPI app in a Docker container on a managed platform. The names of the buttons differ between platforms, but the steps are the same.
- Make sure the app runs in Docker on your machine:
docker buildanddocker run, then call it with curl. - Make the app read its port from the environment if the platform requires it, and listen on
0.0.0.0, not127.0.0.1. - Add a
/healthendpoint that returns 200 when the app is ready. - Push the code, with the Dockerfile and without the
.envfile, to GitHub. - Create a new web service on the platform and connect the repository.
- Add your environment variables (API keys, model name, database URL) in the platform's settings, not in the code.
- Set the health check path to
/health. - Deploy, and watch the build and runtime logs until the service is live.
- Call the public URL with curl and with your test requests.
- Set up automatic deploys from the main branch, so every merged pull request goes live.
Deploying to a virtual machine
When you need a VM, the flow is longer.
- Create the instance and choose its size, region and operating system (usually Ubuntu).
- Create or upload an SSH key, and allow only the ports you need in the firewall or security group: 22 for SSH, and 80 and 443 for web traffic.
- Connect with SSH, install Docker, and clone your repository or pull your image from a registry.
- Put your environment variables in a file on the server with restricted permissions, or better, in the provider's secret manager.
- Run the container with a restart policy, like
--restart unless-stopped, so it comes back after a crash or a reboot. - Put a reverse proxy like Caddy or Nginx in front of it for HTTPS, and point your domain to the server's IP address.
If you also use the cloud provider's command line tool, like the AWS CLI, it asks for an access key and a secret key. Those keys can control your whole account, so create them for a user with only the permissions it needs, and never commit them.
Health checks
A health check is an endpoint the platform calls regularly to know whether your app is alive. If it stops responding, the platform restarts the app or stops sending traffic to it.
Keep it fast and simple. A health check that calls the LLM on every ping costs money and fails whenever the provider has a bad day, which would restart your app for a problem a restart can't fix.
Logs and monitoring
Once the app is deployed, logs are how you see what is happening. Write logs to standard output, and the platform collects them for you.
For an AI application, log at least the request ID, the route, the latency, the model used, the tokens and the errors, as the API page describes, and never log full API keys or sensitive user data. Most managed platforms show logs and basic metrics like CPU, memory and request counts out of the box. As the system grows, tools like Prometheus and Grafana for metrics, and LLM tracing tools for following each request through the model and its tools, become part of LLMOps.
Cost and limits
Two kinds of cost add up for AI apps.
The first is hosting: the platform, the VM, storage and bandwidth. GPU servers are the expensive part, which is why most first projects call a model API instead of hosting a model.
The second is model usage: every request to an LLM API costs tokens, and a public endpoint with no limits can be used by anyone, including bots, as many times as they like.
Before you share a link publicly:
- Set a monthly spending limit or budget alert with your model provider and your cloud provider.
- Add rate limiting per user or per IP address to your API.
- Require a simple key or login for anything beyond a demo.
- Limit the input size, so one request can't send a whole book.
A first-deploy checklist
- The app runs in Docker locally.
- No secrets in the code, the image or the repository.
- Environment variables are set on the platform.
- A
/healthendpoint exists and the platform uses it. - The app listens on
0.0.0.0and the right port. - Logs show each request and each model call.
- Spending limits and rate limits are set.
- You tested the public URL with real requests, including bad ones.
- The README has the live link and explains how to deploy it again.
From the floor
In the AI agents bootcamp, we deployed the customer support agent on Render: the repository already had a Dockerfile, and Render pulled the code from GitHub, built it and ran it. I recommend Render for anyone deploying for the first time, and AWS when the application gets bigger and needs more control.
For the financial analysis agent in the same bootcamp, we deployed to AWS instead: we configured the AWS CLI with an access key, launched an EC2 instance, connected to it with SSH and set up the environment there. It takes more steps than Render, and going through them once shows you what a managed platform does for you.
In another session, we looked at a company asking an online community whether they should move from the cloud to on-premises because they were paying around 31000 euros a year. Other engineers replied that for their small requirements, that cost was low compared with buying hardware and hiring people to run it, and that a hybrid setup or a few changes to their workflow would reduce it. I use that example to show that on-premises is a decision about data sensitivity, steady usage and the team you have, not only about the cloud bill.