Serving a custom YOLOv5 model behind an API, with a React client and CI/CD

There is a point in every computer vision project where the model works. Precision looks fine, the sample images have boxes in the right places, and the notebook is done. That is also the point where most projects stop, because turning the model into something another person can open in a browser is a different job with different problems.

My real-time object detection project is open source and exists mostly to show that second job end to end: a custom-trained YOLOv5 model, a Python API in front of it, a React client that draws detections over a live feed and counts what it sees, and a GitHub to AWS pipeline that deploys the whole thing when I push. The walkthrough video covers the demo; this post covers the decisions.

Draw the boundary at the API

The first decision was the one everything else depends on: the model is a service with an HTTP interface, and nothing else is allowed to import it.

That sounds obvious and it is routinely skipped. It is tempting to load the weights in the same process as whatever serves the page, because it is less code today. The cost arrives when you want a second client, a mobile app or a batch job, or when you want to upgrade the model without touching the UI.

So the contract is small and boring:

POST /detect
Content-Type: image/jpeg

→ 200 application/json
{
  "model": "yolov5s-custom-2024-03",
  "inferenceMs": 23,
  "detections": [
    { "label": "person", "confidence": 0.91, "box": [412, 118, 588, 470] },
    { "label": "bicycle", "confidence": 0.78, "box": [60, 300, 310, 520] }
  ],
  "counts": { "person": 1, "bicycle": 1 },
  "meanConfidence": 0.845
}

Boxes are pixel coordinates in the image that was sent, not normalised, because every client that received normalised boxes had to ask what they were normalised against. The model name travels with every response so a client can tell when the model has changed under it. Counts and mean confidence are computed on the server even though the client could do it, because they are part of what the service means and I want every client to agree on them.

Serving: load once, warm up, say when you are ready

The API is a small Python service. Three details made it behave like a service rather than a script:

1. Load the weights once at startup, in the process, not per request. The first version loaded them lazily on the first call, which meant the first user after every deploy waited ten seconds and assumed the site was broken.
2. Warm up before accepting traffic. The first inference on a fresh process is slow even with weights loaded. The service runs one detection on a bundled image during startup, so /health only reports ready once a real request would be fast.
3. Put the model version in the health response. When a deploy goes wrong you want to know which weights are live without reading logs.

@app.on_event("startup")
def load():
    app.state.model = torch.hub.load("ultralytics/yolov5", "custom", path=WEIGHTS, force_reload=False)
    app.state.model.conf = 0.35
    app.state.model(Image.open("warmup.jpg"))      # first call is the slow one; take it here
    app.state.version = Path(WEIGHTS).stem

Confidence threshold lives on the server too. A client can ask for a lower one per request, but the default is a server decision, because the threshold is part of what "a detection" means for this model.

The client: one request in flight

The React client reads the camera into a canvas, sends a frame, draws the boxes it gets back over the video, and shows a running count per label with the mean confidence. The important rule is that it keeps exactly one request in flight. When a response arrives it sends the next frame. It does not send a frame every 33 ms and hope.

That rule removed the entire class of bugs where boxes arrive out of order and jump around. It also means the client adapts automatically: a fast server gets more frames, a slow one gets fewer, and the drawn boxes are always from the newest response.

Boxes are drawn on a second canvas layered over the video, cleared and redrawn on each response. Drawing into the video frame itself looked the same and made screenshots and recordings messy.

CI/CD: the pipeline is part of the project

The GitHub to AWS pipeline is the part people ask about most, which tells me how rarely ML side projects have one.

On every push to the main branch:

• Tests run. The important ones are not unit tests. They send a known image to the API and assert the labels and approximate box positions. A model that loads but detects nothing passes every unit test.
• A container image is built with the weights baked in, tagged with the commit and the model version.
• The image is pushed and the service on AWS pulls it, restarts, and is checked with /health until it reports ready. If it never does, the previous image stays live.

Baking weights into the image is a choice. It makes images large and makes "deploy new weights" the same operation as "deploy new code", which for a project this size is exactly right. At larger scale you would store weights separately and have the service fetch a pinned version at startup; the health-check and warm-up rules do not change.

What a notebook does not teach you

• The first request after a deploy is a product moment. Warm up.
• Coordinates need a contract. Say which image space the boxes are in.
• One request in flight beats a frame rate.
• An integration test with a real image catches the failures that matter.
• The model's version belongs in every response and in the health check.

The code is on GitHub. The natural next steps are the ones the demo hints at: anomaly detection on CCTV, counting for retail analytics, wildlife monitoring. All of them start from the same boundary, a model behind an API that any client can call.