Context Selection for LLM Apps Without a Vector Store

Most LLM features I’ve seen do the same thing. They fetch everything they know about the user and paste it into the prompt. It is the obvious first move and it is fine at first. Then the token bill climbs, and the answers start wandering, because a model handed a number will find something to do with it. The instinct at that point is to go and rewrite the prompt. I spent a while doing that before admitting the prompt was not the part that was wrong.

Someone came to me with that problem in their coaching app. Every answer had to be personalised from a handful of backend services, the prompt had grown into a dump of everything they knew about the athlete, and it was getting expensive and vague at the same time. What they needed was the other approach: a layer whose only job is to decide what the model is allowed to see, and which can explain that decision afterwards without calling the model at all.

Four services hold everything the app knows about an athlete: their account and preferences, their slow-moving baselines like aerobic capacity and mobility, a readiness signal generated every morning for training, recovery, nutrition and injury, and the day’s weather and air quality, which is one payload per city that everyone in it shares. A question comes in, the layer fetches what it needs, and an LLM writes the reply in the athlete’s language at a length that suits the question. It coaches. It does not diagnose, and anything that sounds clinical is supposed to end with go and see someone.

If you have read anything about getting context into a prompt, the answer was probably retrieval. Embed the corpus, search by similarity, take the top few. There is no corpus here. Four services, about a dozen fields between them, all current, all about one person. Similarity has nothing to grip on. Whether this morning’s injury reading belongs in the prompt depends on what the athlete asked about, and that is a rules problem.

Everything worth talking about is in the middle step.

What the model gets to see

I broke the four payloads into eleven small pieces. Each one knows how to pull a single fact and write it as one line: the athlete’s current training block, their mobility and strength baselines, this morning’s injury readiness, their chronotype, today’s air quality. Keeping them small is the point. You cannot choose carefully between four blobs of JSON, but you can choose carefully between eleven lines.

I kept the picking in a table instead of a pile of if-statements. Every kind of question the app supports gets an entry in a YAML file a coach can read without knowing any Python:

intents:
  training:
    primary: [readiness_training, baseline_aerobic, training_block]
    secondary: [conditions_today, athlete_profile]
    exclude: [readiness_recovery, readiness_injury]
    max_words: 250

  injury:
    primary: [readiness_injury, baseline_mobility, baseline_strength]
    secondary: [training_block, athlete_profile, conditions_today]
    exclude: [readiness_nutrition, readiness_training]
    max_words: 220

primary always goes in. secondary goes in if there is room. exclude is the one that ends up mattering: it names context that is wrong for this kind of question and should be withheld even though it is sitting right there in memory. It exists because early prompts kept wandering. Ask about a training block, get a paragraph about your sleep score.

I did not put this in a file because YAML is pleasant. Which data answers which question is something a coach knows and I do not, and the moment I bury it in branches, the only person who can check it is me. In a file you can diff it and argue with it in review, and the app validates it on startup, so a typo or a wrong field name kills the boot instead of one unlucky request at three in the morning.

What a file does not do is hand the rules to the coach. Editing it still means a pull request and a deploy, so the person who actually holds the knowledge is still going through an engineer to apply it. If you want them changing it themselves, the file is the first half of the job and a versioned table behind a small admin screen is the second, with the checked-in file staying as the default. I stopped at the first half.

The table alone cannot tell today from next quarter, so a second pass runs after it, keyed to how far ahead the question looks. “Today”, “this week” and “over the next few months” want different context out of the same topic, so a set of modifiers promote, demote and drop pieces before anything else runs. A same-day question pulls today’s conditions up into primary and pushes the long-run baselines down. A multi-month question drops conditions entirely, since today’s air quality says nothing about a twelve-week block, and promotes the training block itself. Every modifier carries its reason as a string, and that string travels all the way into the response.

Then I check what actually came back. I did not want one missing flag covering everything, because “the service timed out” and “the service answered and had nothing for this athlete” send you looking in completely different places. So it records three separate reasons: it could not reach the service, nobody has ever computed that value for this athlete, or the payload arrived fine with the field empty. You get the reason by name.

The budget comes last, and it only touches secondary context. Primary is what the answer stands on, and I did not want it disappearing quietly to save a few tokens. Each piece carries a token cost in the registry: twelve for the athlete profile, thirty for the training block, twenty-five for a baseline, forty-five for a readiness signal, forty for today’s conditions. Secondary pieces are taken in order until the budget runs out. Those costs are hand-assigned rather than measured, and the paid tier’s budget of 1400 never actually binds at realistic payload sizes, so the only budget doing observable work is the free tier’s 50. The paid number is not doing anything yet, which is worth saying rather than letting it sit there looking load bearing.

Against a naive prompt that dumps the whole fetched bundle in as JSON, built with the same system block so the comparison is like for like, selection came out 37.7% smaller. 259 tokens against 416. Those are small payloads, so treat the percentage as a measurement of the method rather than a number to quote at your own traffic. The gap widens as the bundle grows, which is the direction real payloads go.

None of this touches the network. No async, no model, just functions over data I had already fetched, which also let me expose it read-only: ask what it would pick for a question and it tells you without generating a word.

flowchart TD
    Q["Question"] --> S["score_intents<br/>weighted keyword sum"]
    S --> C{"top >= 1.0 and<br/>margin >= 1.0 ?"}
    C -->|yes| R["intent = top scorer<br/>source = RULES"]
    C -->|no| L["ask the model<br/>source = RULES_INCONCLUSIVE"]
    R --> P["policy lookup<br/>primary / secondary / exclude"]
    L --> P
    P --> X["apply exclude list<br/>runs the same either way"]
    X --> M["scope modifier<br/>promote / demote / drop"]
    M --> A["drop what upstreams<br/>could not supply"]
    A --> B["spend the budget<br/>on secondary only"]
    B --> O["selected context"]

When two intents tie

All of that starts with knowing what the question is about, which is the top left of that diagram. I did not want a model call on the hot path for something as small as “is this about training or about an injury”, so the first attempt is a weighted keyword sum:

_INTENT_TERMS: dict[Intent, dict[str, float]] = {
    Intent.TRAINING: {
        "train": 2.0, "workout": 2.0, "intervals": 2.0, "mileage": 2.0,
        "race": 1.5, "tempo": 1.5, "session": 1.0, "pace": 1.0,
    },
    Intent.INJURY: {
        "pain": 2.0, "injury": 2.0, "strain": 2.0, "tendon": 2.0,
        "hurt": 2.0, "physio": 1.5, "niggle": 1.5, "rehab": 1.5,
    },
    ...
}

Sum the weight of every term the question contains, per topic. Plurals count, via a + "s" check that is exactly as sophisticated as it looks. The winner has to clear two bars before it is trusted:

confident = (
    # Zero on-domain signal never takes the fast path, whatever the ranking.
    top_score > 0
    and top_score >= CONFIDENT_THRESHOLD   # 1.0
    and top_score - runner_up_score >= MARGIN   # 1.0
)

That first clause does nothing while the threshold sits at 1.0, since anything clearing 1.0 is already above zero. It is there so that lowering the threshold later cannot quietly hand the fast path to a question that matched no vocabulary at all.

Clear both and the rules decide, with no model involved. Fail either one and the question goes to the LLM with a small classification prompt, and the result is tagged as having come from the model rather than from the rules. I liked this part. The common case is free and fast, and the system writes down when it does not know.

A trained classifier would beat a bag of words at this, and if there were logged questions to learn from I would have used one. There were none. The other reason to keep it as a vocabulary is that it stays readable by the same person who owns the policy file, so adding a term is a one-line change they can argue with.

Then a coaching app gets asked a six-word question.

Should I train through this pain?

train is there and carries 2.0 for training. pain is there and carries 2.0 for injury. Nothing else in those six words appears in any vocabulary. Training scores 2.0, injury scores 2.0, and the margin is 0.0, so the rules step aside and the model labels it one or the other.

Say it comes back as training. Look at what training excludes.

readiness_injury is on that list. The athlete asked whether to train through pain, the classifier could not tell which half of the sentence was the question, and the policy then removed the injury signal from the prompt on purpose. Flip the coin the other way and injury wins, which excludes readiness_training, and now the model is reasoning about a race build without being told there is a race build. Both branches drop the half of the question the athlete cared about.

The answer that comes back does not look wrong. Confidence is computed from how much of the requested context arrived rather than asked of the model, and all of it arrived, so confidence is high. The source list is literally the labels of what went into the prompt, so it cannot name something that was never there. Both of those are working exactly as designed, and both are describing a prompt that was assembled wrong.

The classifier knew it was guessing and recorded it. The policy knew which context to withhold. The selection pass knew and reported every reason a piece was dropped. Each of the three held part of the answer, and the uncertainty existed in the system the whole time. It just never reached the code that needed it.

The same gap has a second door. The tokenizer behind the scoring is re.compile(r"[a-z]+"), so a question written in any script that is not Latin scores zero against every term in every vocabulary. Not low. Zero. An entire language routes to the model every single time by construction. I had written that up as a feature, on the grounds that unfamiliar input belongs with the model anyway. That is true, and it is also a rationalisation, because I did not choose it. It fell out of a regex and I decided afterwards that the consequence was acceptable.

What I would change

Withholding context is a strong move and it only makes sense when the routing is certain. Mine was sometimes certain and sometimes a coin flip, and the exclude list ran identically in both cases.

The fix is small, which is the annoying part. The selection function already receives where the intent came from, and uses it only for reporting. When the intent came from the model rather than the rules, the exclude list should soften: keep the winning topic’s primary context, and stop actively deleting the runner up’s signal when the runner up was tied. Better still, when two topics land within a point of each other, take both of their primaries and let the budget sort out what fits. That sends more context than a clean classification would, and it is the right trade, because a few extra tokens cost less than answering a question about pain with the pain data deliberately removed.

None of this was hard to find once I went looking, because every drop carries a typed reason and the service records where each intent came from. What I had not built was anything that reads those two facts together. The log told me the intent came from the model, the policy file told me what that intent withholds, and I was the one who had to put them side by side. A pipeline that could not describe its own decisions at all would have produced the same answer, and I would still think it was working.