LLM Response Cache in Python: Exact Match with SQLite, Then a Semantic Threshold

You can cut an LLM bill by not calling the model for questions you have already answered. The catch is the word "already". An exact-match cache is safe and boring. A semantic cache will happily return Alice's subscription plan to Bob if you let it. This post builds both in Python, runs them on a small synthetic query set, and shows the threshold sweep that decides whether your cache is helping or lying.

If you want the Redis and GPTCache route, semantic caching with Redis and GPTCache covers it. This post is narrower: what goes in the key, how to choose a threshold with evidence, and what not to cache. For savings on the provider side, which is a different mechanism, see Claude API prompt caching.

What I ran, and what is fake

Everything below ran on 2026-10-23 in Python 3.13 with numpy and fastembed. Three things to know before you read the numbers:

• The data is synthetic. I wrote 10 cached questions and 26 test queries for an invented SaaS product, and labelled each with the intent it should map to. Real traffic will look different.
• No LLM was called. The "model" is a function that returns ANSWER[intent], so I can grade whether a cache hit returned the right answer. Model latency in the averages below is a simulated constant of 800 ms.
• The embeddings are real. I used BAAI/bge-small-en-v1.5 through fastembed on CPU. That is one model; yours will score differently.

How should an exact-match cache key work?

Hash everything that can change the answer. A prompt string alone is not enough.

import hashlib, json, sqlite3, time

def cache_key(model, system, prompt, params, scope=""):
    blob = json.dumps([model, system, prompt, params, scope],
                      sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()

class ExactCache:
    def __init__(self, path=":memory:", ttl=3600):
        self.db = sqlite3.connect(path)
        self.db.execute("CREATE TABLE IF NOT EXISTS c (k TEXT PRIMARY KEY, v TEXT, created REAL)")
        self.ttl = ttl

    def get(self, key):
        row = self.db.execute("SELECT v, created FROM c WHERE k=?", (key,)).fetchone()
        if row and time.time() - row[1] < self.ttl:
            return row[0]
        return None

    def put(self, key, value):
        self.db.execute("INSERT OR REPLACE INTO c VALUES (?,?,?)", (key, value, time.time()))
        self.db.commit()

SQLite is plenty for one process and a few hundred thousand rows. Use a file path instead of :memory: to survive restarts. The scope field is the one people forget: put the user or tenant id there whenever the answer can differ per person.

An exact cache only hits on byte-identical requests, so it earns its keep on repeated automated calls (the same classification prompt, a retried job, an FAQ button) and rarely on free-typed chat.

I ran the 26 test queries through it twice, once with the scope left out of the key and once with the user in it:

scope in key=False hits=6/26 wrong_answers_served=3
scope in key=True  hits=4/26 wrong_answers_served=1

Without scope, six queries matched a stored string exactly, and three of those were wrong: Bob received Alice's answers to "What is my current plan?" and "When does my subscription renew?", and the status question returned a stale answer. Scoping fixed Bob. The remaining wrong answer is "Is the service down right now?", which is time-dependent. A scope cannot fix that. A short TTL or a bypass can.

How does a semantic cache work?

Embed the query, compare it with the embeddings of cached queries, and return the stored answer if the best cosine similarity clears a threshold. With normalised vectors that is one matrix-vector product.

import numpy as np

class SemanticCache:
    def __init__(self, embed, threshold=0.9):
        self.embed, self.threshold = embed, threshold
        self.vecs, self.items = np.zeros((0, 0)), []

    def get(self, text, scope=""):
        if not self.items:
            return None, 0.0
        scores = self.vecs @ self.embed([text])[0]
        for i, it in enumerate(self.items):       # only same-scope entries may match
            if it["scope"] != scope:
                scores[i] = -1
        best = int(np.argmax(scores))
        if scores[best] >= self.threshold:
            return self.items[best], float(scores[best])
        return None, float(scores[best])

    def put(self, text, value, scope="", meta=None):
        v = self.embed([text])
        self.vecs = v if not self.items else np.vstack([self.vecs, v])
        self.items.append({"text": text, "value": value, "scope": scope, "meta": meta})

This is a linear scan, fine for thousands of entries and wrong for millions. If you outgrow it, you need a vector index. The embedding step also costs time: my mean lookup was about 20 ms on CPU, almost all of it embedding the query. A cache hit is cheap compared with an LLM call, but it is not free.

The 26 test queries break into four kinds: 12 paraphrases of a cached question (should hit), 8 near misses with similar wording but a different question (should miss), 4 personal questions, and 2 time-dependent ones.

What threshold should a semantic cache use?

I swept the threshold and graded every hit. "Wrong" means the cache returned an answer for a different intent than the query's.

thr   hits  correct  wrong  hit_rate  lookup_ms  avg_latency_ms(sim)
0.95     9        6      3       35%       19.5         543   wrong by kind: {'personal': 2, 'fresh': 1}
0.90    15       11      4       58%       23.5         362   wrong by kind: {'near': 1, 'personal': 2, 'fresh': 1}
0.85    18       12      6       69%       20.3         266   wrong by kind: {'near': 3, 'personal': 2, 'fresh': 1}
0.80    21       13      8       81%       22.4         176   wrong by kind: {'near': 5, 'personal': 2, 'fresh': 1}
0.75    23       14      9       88%       21.3         114   wrong by kind: {'near': 6, 'personal': 2, 'fresh': 1}
0.70    23       14      9       88%       33.8         126   wrong by kind: {'near': 6, 'personal': 2, 'fresh': 1}

The avg_latency_ms(sim) column assumes an 800 ms model call on every miss, so it only shows the shape: more hits, lower average. The 33.8 ms lookup at 0.70 is noise from a single run.

Look at the wrong column. Even at a strict 0.95 threshold, three answers were wrong. They are the personal and time-dependent queries, which score 1.000 against an identical stored string. No threshold stops that, because the text is the same and the right answer is not.

Next, the raw best scores for the paraphrases and the near misses:

0.980 same  para     What are the steps to cancel my subscription?
0.794 same  para     i want to cancel my plan, how
0.913 same  para     How can I stop my subscription from renewing?
0.959 same  para     Can I temporarily pause my subscription?
0.949 same  para     How do I forget and reset my password?
0.903 same  para     I forgot my password, what now?
0.941 same  para     Is there a discount for students?
0.847 same  para     Do students get a lower price?
0.908 same  para     Can I download my data as a CSV file?
0.952 same  para     How do I get a PDF export of my data?
0.898 same  para     What cards or payment methods can I use?
0.624 same  para     Do you take UPI?
0.906 DIFF  near     How do I delete my subscription and my account?
0.822 DIFF  near     How do I upgrade my subscription?
0.663 DIFF  near     How do I change my email address?
0.797 DIFF  near     How do I reset my two-factor authentication?
0.804 DIFF  near     Do you offer a nonprofit discount?
0.886 DIFF  near     How do I export my data as JSON?
0.688 DIFF  near     Which countries do you ship to?
0.881 DIFF  near     How do I import data from CSV?

The paraphrases range from 0.624 to 0.980 and the near misses from 0.663 to 0.906. The ranges overlap. "How do I delete my subscription and my account?" scored 0.906, higher than five legitimate paraphrases. Counting from this table, a threshold of 0.92 serves zero near-miss answers but hits only 5 of the 12 paraphrases. A threshold of 0.85 hits 9 of 12 and serves 3 wrong answers.

My reading: on this data, a threshold alone cannot separate "same question" from "similar question". Two honest options follow. Pick a high threshold and accept a low hit rate, or keep a lower threshold and add a second check before serving a hit, such as comparing extracted entities or asking a cheap model to confirm. I did not test the second option.

Do not copy 0.9 from this post. It came from 36 hand-written questions and one embedding model. The method transfers: label 50 or so real paraphrase pairs and near misses from your logs, print the scores, and decide how many wrong answers you can live with.

When is caching dangerous?

Three situations produced wrong answers above, and each has a guard.

Personal answers. Scope the cache by user, as in the scope argument. Better still, do not cache anything where the answer comes from the user's own data. A cache keyed per user has a tiny hit rate anyway.

Time-sensitive answers. "Is the service down right now?" has no safe cached form. Use a very short TTL or bypass the cache for those request types. My bypass rule below is a crude regex, which is fine for a demo and fragile in production:

FRESH = re.compile(r"\b(right now|today|currently|latest|current status)\b", re.I)

Near-miss questions. Handled by the threshold and any extra verification, as above.

With user scoping plus the bypass rule at 0.85, on the same data:

hits=14 wrong=3 bypassed_by_rule=2 of 26

Personal and time-dependent wrong answers went to zero. The three remaining wrong answers are the near misses at 0.906, 0.886 and 0.881. The guard fixed the categories that a threshold cannot fix and left the one it can.

Also think about the sensitive part: a cache stores model output, and sometimes the prompt. Do not put personal data in a shared cache, and set a TTL so old answers expire when your product changes. Include a version string in the key (the model name, a prompt version) so a deploy invalidates stale entries.

Exact or semantic: which one first?

Start with the exact cache. It has no wrong-answer risk beyond the key design, and it handles retries and repeated automated prompts. Add the semantic layer only when your logs show many near-duplicate free-text questions, your questions have stable answers, and you have a labelled set to tune against. For choosing an embedding model, the embedding models comparison is a starting point, and the Claude bill checklist shows where response caching sits next to batching and routing.

Checklist before you ship a cache

• Key includes model, system prompt, prompt, parameters, tools and a prompt version.
• Personal answers are scoped by user or never cached.
• Time-dependent request types bypass the cache or use a short TTL.
• Semantic threshold chosen from your own labelled paraphrases and near misses, with the wrong-hit rate printed.
• You log hit, miss, score and which stored query matched, so you can audit a bad answer.
• You can clear the cache in one command.

What I did not verify

I did not test with a real LLM, a large cache, concurrent writers or a vector index. The query set is synthetic and small, so the percentages are illustrations of the method and not benchmarks. I used one embedding model, and the scores will move with a different one.