StreamForge

Context

Magnifi accepts video in most of the formats a broadcaster is likely to have: HLS, SRT, RTMP, MP4. That covers almost everyone.

One broadcaster it didn’t cover. They ran cameras in stadiums, and those cameras streamed straight into Mimir, their media asset manager. Their video already had a destination, and it wasn’t us. To send us a stream in a format we accepted, they would have had to add equipment and licensing on their side, on top of what they were already paying us. They came back with the obvious question instead: the video is already sitting in our asset manager, can you just pull it from there?

Mimir had no playlist export. It held the segments. It would not hand us a manifest that pointed at them.

So the format gap wasn’t really a format gap. They had the video and we could read it. What was missing was the file that says what order to play it in.

Constraints

Three parties, and I only controlled one. Us, the broadcaster, and Mimir. Every question about behaviour had to travel across two org boundaries before it came back.

The broadcaster wasn’t going to change much. They had their own internal reasons and their own products depending on the current setup. I could ask for an event on their side. I could not ask them to restructure how their recording worked.

Match schedules are lumpy. There are days with several games in parallel and days with nothing at all. Whatever I built had to cost nothing when nothing was happening.

It had to land in the existing pipeline. No special path downstream. Whatever came out the other end had to be ingestible exactly like any other HLS source.

Design

They publish a recording-start event to SNS. We subscribe.

On that event, StreamForge spins up one Fargate container for that stream, configured from a mix of what the event carries and what we hold in the database for that partner. The container polls Mimir’s API for new segments, pulls the raw .ts files into our bucket, and after each one, rewrites the HLS playlist to include it.

That playlist is a live file. Our media pipeline ingests it while the container is still appending to it, exactly the way it would treat any other HLS input. Nothing downstream knows the stream arrived this way.

flowchart TD
  CAM["Stadium cameras"] --> MAM["Mimir<br>(their asset manager)"]
  MAM -->|"recording start"| SNS{{"SNS"}}
  SNS --> SF["StreamForge"]
  CFG[("Partner config")] --> SF
  SF -->|"one per stream"| FG["Fargate container"]
  FG -->|"poll for new segments"| MAM
  FG -->|"raw .ts"| S3[("S3 bucket")]
  FG -->|"append"| PL["Live HLS playlist"]
  PL --> PIPE["Magnifi media pipeline"]

The playlist is being written and read at the same time. That’s the whole trick.

Missing segments. A segment can arrive late, out of order, or never. The container retries with exponential backoff and jitter. If it still can’t get the segment, it writes a black frame for that duration and keeps going.

Black frame rather than a discontinuity marker, and that choice took some working out. The obvious move is EXT-X-DISCONTINUITY, which is what the tag exists for. But this feed carries data markers that ride along with the real-time stream, and a discontinuity breaks the timeline those markers are positioned against. A black frame holds the duration, so everything downstream still lines up. A gap in the picture is cheaper than a timeline that no longer matches the data.

Tradeoffs

Fargate per stream over a long-running service. I can’t predict when this needs to run. There are days with several parallel matches and days with none, so a service sitting idle is money spent on nothing. Containers also gave me the scaling answer for free: parallel streams means more containers, and there’s no pool to size or drain. A long-running service would have made both of those my problem.

Pulling from Mimir over asking them to push. They could have pushed us a format we already accepted. It would have cost them hardware and licensing, and they’d already told us that cost was the thing standing between them and onboarding. Building the puller was cheaper than losing the deal.

Black frames over discontinuity markers. Covered above. The tag designed for the job was the wrong call here because of what else was riding on the timeline.

Outcome

It ran at 50+ concurrent streams and scale was never the interesting part. Containers are cheap and the work is embarrassingly parallel.

The hard part was the integration. Three parties, one of them a vendor neither of us controlled, and a broadcaster who tested by running the thing all day rather than by sending us a test plan. That turned out to be the right way to test it. Containers crashed on cases I hadn’t imagined.

Two of them were about lifecycle rather than throughput, which is where I should have been looking from the start. In one, ingestion stopped on their side mid-stream and my container kept politely asking for segments that were never coming. In the other, the stream ended but the recording-stop event never arrived, so nothing told the container to shut down.

Both are the same bug in different clothes: I had built a container whose lifetime depended on somebody else’s event, and I had trusted that event to always show up.