The Partner Layer

Context

When I joined, the code that sent metadata to partners lived inside the clip generation logic. Not next to it. Inside it. There was no separate service, no versioning on the endpoints partners called, and no retry when a delivery failed. The endpoints were internal ones that partners had been pointed at.

That coupling had a predictable effect. Someone would change clip generation for an internal reason and a partner’s integration would break. When it broke, the error a partner saw told them nothing. Companies don’t mind failures. They mind not knowing what failed.

One partner felt this most. They consumed our metadata to drive their own workflow, and they were serving clients on the other side of it, so our failures became their failures. Their only channel to us was Slack. During live sport, that meant they pinged us mid-match, sometimes called, and more than once we sat in an RCA call explaining an outage in a system they couldn’t see into. Those calls are what convinced me to stop patching.

So I went to the partners and asked what actually hurt. The answers were consistent. Errors were unreadable. Delivery was unreliable. When something broke, the fix took days. I threw the old path away and started over.

Constraints

The old path stayed live. Partners were mid-season. Nothing could be cut over on my schedule, so the new service ran alongside the old one until every partner had moved. That took about three months, after which nothing new was ever built on the old path.

I don’t control partner code. Once an endpoint is published and someone has integrated against it, changing it costs them a deploy cycle I have no visibility into. That pushed the design toward a small surface, defined before it was built rather than grown by accretion, and toward pushing data to partners rather than letting them ask for it.

Internal APIs had to stay internal. They carried things we couldn’t expose. Fixing them in place was never on the table.

Two people. Me and one junior engineer who did a large share of the work. The team grew to three later.

Five months to something usable. The rebuild started in June 2024. Partners could work against it by November. The three-month migration off the old path came after that.

Design

Partners get an access key and secret, plus documentation. Before any stream, they load the context we need to produce correct metadata: tournaments, teams, players, match schedules. Each of those has its own endpoints on the partner service, so a partner ingests schedules through one and roster data through another.

Then they call stream create, and hand us a webhook URL along with it.

flowchart TD
  P["Partner"] -->|key + secret| API["Partner API"]
  API --> SCH[("Tournaments, teams,<br>players, schedules")]
  API -->|"stream create + webhook URL"| BE["Backend"]
  BE --> MEDIA["Media service"]
  BE --> AI["AI service"]
  MEDIA -->|"clip ready"| SNS{{"SNS"}}
  AI -->|"metadata ready"| SNS
  SNS --> PS["Partner service"]
  PS -->|"is this stream ours?"| DB[("Stream + org records")]
  PS --> MAP["Schema orchestration"]
  MAP -->|"signed payload"| WH["Partner webhook"]
  WH -.->|"5 attempts, exponential backoff"| PS
  PS --> AUDIT[("Delivery audit table")]

The partner service is a consumer, not a step in the media pipeline. Nothing upstream knows it exists.

From there the media service produces clips and the AI service produces metadata, and both fan out over SNS. The partner service consumes those events and decides whether each one is its business: it checks the organization ID on the event and looks up whether that stream was created by a partner who is expecting delivery. Most events aren’t. Those get dropped.

For the ones that are, it fetches what it needs and runs the payload through an orchestration layer that maps our baseline schema into that partner’s shape.

The baseline schema is frozen. Partners take it as given. Inside it, a defined subset of fields can be overridden per partner, and that override is configuration rather than code. Adding a new partner’s shape takes about an hour and no deploy.

Delivery is five attempts with exponential backoff. Each payload is signed with a secret shared with that partner and the signature travels in a header, so they can verify the call came from us. Some partners also wanted their own credentials echoed in headers so they could authenticate on their side, which the same mechanism carries.

Ordering isn’t guaranteed. A partner can in principle receive events out of sequence. Given the shape of this data that has almost never mattered, and I’d rather be honest that it’s possible than claim a guarantee the system doesn’t make.

Retries and responses go into a dedicated table. Before, this was log output that someone had to go read. Now every attempt, every response code, and every error body is queryable, which is what makes a partner’s question answerable in minutes instead of an RCA call.

Tradeoffs

Webhooks over polling. We know when a clip is ready. Polling would have meant partners asking us repeatedly whether anything had happened, most of those calls returning nothing, and all of them hitting the database. The party that knows should be the party that speaks.

A separate service over fixing the internal APIs. The internal APIs carry things partners can’t see, and they exist to serve our own product. The partner service shares business logic with them but sits behind its own contract, so an internal refactor stops being a partner-facing incident.

A frozen schema over per-partner code. Custom code per partner is what the old path effectively was, and it doesn’t scale past a couple of partners: every request becomes a deploy, and no team has that bandwidth. Freezing the schema moved the negotiation earlier. Partners were fine with it, partly because a contract that won’t change is worth something to them too.

What I got wrong. In the first version I sent metadata only once the full clip was ready, video included. That felt correct and it wasn’t. The media service takes time to process video, and the metadata is usually ready well before it, so partners were waiting on something they hadn’t asked for. Decoupling the two took about two weeks.

Outcome

Onboarding went from around a week to about a day, but the number understates what changed. The week wasn’t engineering time. It was a partner telling us they wanted custom data, describing a shape nobody had constrained, our writing code for that shape, and every partner arriving at a different implementation because nothing stopped them. With a frozen schema, the conversation is which fields to override, and that’s configuration.

The service handles 300K+ events a day and the work fed into more than $1M in cumulative partner deals.

What I hadn’t planned for. One partner started running six or seven live games in parallel. Nothing fell over, but two gaps showed up. A different service was returning incorrect data, and I had no way to force a re-send once it was fixed, so I built one. Then the re-sends became their own problem: the partner told us we were storming their server with repeats of the same payload.

The fix was a check in Redis. Hash the payload, and if an identical one went to the same destination inside the last sixty seconds, drop it. Cheap, and it ended the problem.