Building a Local-First Search Engine for CCTV Footage

If you have a camera pointed at a room and you’ve ever lost your keys in that room, you know how it goes. You open the app, drag the timeline back a few hours and scrub. Maybe the app flagged some motion. You tap through each clip until you see yourself walk in with the keys, and then you watch where your hands go.

The footage has the answer. You just can’t ask it anything.

That bugged me enough to start a project about it. It’s called Triloch. You point it at hours of recorded video, ask “where did I leave my keys?” or “when did the package arrive?”, and get back the right few seconds of video with a reason it matched. Everything runs on machines in my house.

This is the first post in a series I’ll keep writing as it gets built.

Why this one

Most of what I’ve enjoyed building this year has come down to one problem: finding the right thing in a pile of stuff and handing only that to whoever needs it. Samvad’s retrieval was that. So was a context-selection engine I wrote about last week. Both worked on text, where the tools are well worn. You can embed it, search it, filter it and rank it.

Video gives you none of that for free. An hour of footage is a long list of pictures with a timestamp on each. Before anything can be searched, something has to watch it and decide what happened, and that part is new to me. I wanted a project where I couldn’t lean on what I already knew.

It also had to be something I’d actually use. There’s a camera in my room already.

What I’m building

The obvious move in 2026 is to hand the footage to a vision model and let it describe what it sees. The numbers stop you pretty quickly. One camera at 25 frames a second produces about 2.2 million frames a day. No laptop runs a vision model over all of that, and sending it to a paid API means paying every month to upload video of my bedroom to someone else’s server. I didn’t want either.

Most of those frames are useless anyway. An empty room at 3 AM looks the same for four hours straight. So the plan is to split the work by how much it costs:

flowchart TD
    CAM["Cameras<br/>one indoor,<br/>two public traffic"] --> REC["Recorder<br/>keeps the footage<br/>and a clean index"]
    REC --> GATE["Cheap filter<br/>watches everything,<br/>flags movement"]
    GATE --> VLM["Local vision model<br/>describes only<br/>the flagged moments"]
    VLM --> IDX["Search index<br/>over those<br/>descriptions"]
    IDX --> Q["Question in,<br/>video clip out"]
    classDef planned stroke-dasharray: 5 5
    class VLM,IDX,Q planned

The solid boxes run today. The dashed ones are next.

The search box at the bottom is the part I care about most, and it’s where the hard question lives. When a clip gets described and indexed, I have no idea what anyone will ask about it later. “Show me the dog on the sofa” can probably be answered by matching a description. “Where did I leave my keys?” can’t, because the answer is spread across several moments: I walked in holding them, went to the desk, put something down, left. Whether that needs something fancier, like a graph of objects and places, or whether plain good search gets there anyway, is something I’d rather test than guess.

So the one rule I’ve set myself is to measure before getting clever. Early on I’ll build a set of questions with the right answers marked by hand, and anything sophisticated has to beat a simpler version on that set to stay in.

Where it is today

A few days in, the first two boxes work.

The recorder runs on my M1 MacBook. It records the camera in my room plus two public traffic cameras from Caltrans, California’s transport department, because my room will never show me rain, headlights or night traffic. It keeps a clean list of every five-minute chunk it has saved, and it deletes the oldest footage once it hits 20 GB, since the laptop doesn’t have much disk to spare. Getting this solid took longer than I expected. The camera’s own clock turned out to drift away from the laptop’s by several seconds, which quietly broke my first design.

The cheap filter works too. It looks for pixels that change, and on the first real run it kept about a quarter of the room footage and threw away the rest. On the traffic cameras it kept 81 to 93%, because a highway never stops moving. Fine by me. Better to find out now than after building everything else on top of it.

It also has a blind spot I watched happen live. When I sat still and read a book, the filter slowly decided I was part of the furniture. A moment it drops can never be found by search, so that matters more than a false alarm. Measuring how much it matters, against footage I’ve labelled by hand, is what I’m working on now.

The vision model comes after that. It’ll run on my other MacBook, an M3 Pro, and I haven’t picked the model yet.

Following along

The code is private while I work through the design, but I’m writing everything down as I go. The next post is probably that clock.

If you’ve built anything that searches video, or you’ve tried and given up, I’d love to hear what broke first.