Getting My CCTV Captions Ready for Search

Getting My CCTV Captions Ready for Search

by Anindya Chowdhury

In part 4, I spent two weeks testing models on my home CCTV footage and couldn’t pick a clean winner. Marlin caught actions Qwen missed, Qwen noticed details Marlin missed, and both invented things. I kept them around and went back to building Triloch, my local CCTV search engine.

The search box I promised now exists. It returns up to three matching clips and plays them beside their captions, using the original September Marlin library. The new processing work hasn’t replaced those captions yet.

For that work, I gave the two models different jobs. Marlin watches the video and writes a description. Qwen reads that description and picks out claims I can later put into search. The final check still involves me watching myself in my office, which turned out to be necessary even when both model calls finished successfully.

Eight seconds, 275 tokens

Marlin hit the limit I’d set while describing an eight-second clip. I’d allowed 256 tokens, the pieces of text a model generates. It used them all without reaching its end marker, so I still had an unfinished reply.

I raised the ceiling to 512 in a new processing recipe. A later attempt finished at 275 tokens. For eight seconds of footage, Marlin had plenty to say.

The earlier reply stayed saved as truncated. I didn’t want a later step treating an unfinished description as the complete account of a clip just because the text looked usable. The successful reply got its own record, with the original wording preserved.

That record also connects the caption to its source recording, the sixteen prepared frames, the inputs actually submitted to the model, and the prompt and code used for the attempt. I can follow a sentence back to what produced it without asking Marlin to describe the scene again.

Qwen gets a different job

With a complete caption saved, I pass its text to a separate local Qwen call. This time Qwen gets no video. Its task is to select statements from Marlin’s description and return them as structured records for later indexing.

Two statements in the completed caption were:

The man stands up and walks toward the door.

The man opens the wooden door.

Qwen selected both among six claims from that clip. Each selected claim has to quote an exact passage from the caption, and my parser checks where that passage appears. The saved claim links back to both model replies.

That means I can check the extraction without running the vision model again. If the parser rejects a reply, the original text survives. If I change the parser later, I can try it against that same saved response. What this check establishes is that a claim came from Marlin’s words. I still have to compare those words with the footage.

The door was already open

When I watched that clip beside the caption and extracted claims, I was already standing, hunched over the table while working. I straightened and walked away. There was no getting up from a chair.

The door was already open too. I pushed it further open on my way out. Marlin had described the broad sequence while getting the starting state wrong twice, and Qwen had carried both statements into the records.

I had a complete reply and six extracted claims, with those two mistakes still inside them. Qwen couldn’t supply a second visual check because I’d only given it text.

A claim can trace back to a mistaken caption The saved model caption says “The man opens the wooden door.” The extracted claim preserves that same sentence and links back to the caption text. Extraction does not independently verify the video. In a separate owner review, the door was already open and the owner pushed it further open. The saved caption and extracted claim remain unchanged. SAVED CAPTION“The man opens the wooden door.”MODEL CLAIM EXTRACTED CLAIM“The man opens the wooden door.”MODEL CLAIM Extraction links to caption text. It does not verify the video. OWNER REVIEWThe door was already open. I pushed it further open.OWNER’S ACCOUNT · PARAPHRASED The caption and extracted claim remain unchanged. A claim can trace back to a mistaken caption The saved model caption says “The man opens the wooden door.” The extracted claim preserves that same sentence and links back to the caption text. Extraction does not independently verify the video. In a separate owner review, the door was already open and the owner pushed it further open. The saved caption and extracted claim remain unchanged. SAVED CAPTION“The man opens the wooden door.”MODEL CLAIM EXTRACTED CLAIM“The man opens the wooden door.”MODEL CLAIM Extraction links to caption text. It does not verify the video. OWNER REVIEWThe door was already open. I pushed it further open.OWNER’S ACCOUNT · PARAPHRASED The caption and extracted claim remain unchanged.
The saved caption survives extraction unchanged. My review stays separate. Quoting a caption does not establish whether it describes the video correctly.

I saved my corrections beside the models’ account, linked to the original response. Rewriting that response would have hidden the mistake I was trying to understand. I want to see what Marlin said, what Qwen selected, and what I found when I watched the clip.

What I can build on

If I eventually ask when someone got up from a chair, that standing claim could send search toward the wrong event. I haven’t run that query on these new records yet. When I do, I can inspect the returned claim and play the clip instead of trying to judge the answer from a plausible sentence alone.

That’s what the new path gives me before I add these records to search. It preserves the model’s account, including failed attempts, and gives me somewhere to attach my own review. I can change how I extract claims while keeping the original descriptions available for comparison.

So far, two development clips have completed generation and extraction across separate runs. They don’t establish general accuracy, complete extraction or continuous processing speed. I’ve kept the known errors and left October 2 held out instead of tuning around these examples.

Next I’ll work on indexing these records and connecting search results to their evidence, while the existing box continues to search the September captions.

Full size image