A recent study found that large language models often favor established tools. In one web-server experiment, Flask appeared far more often than FastAPI. That raises a practical question for anyone shipping a newer technology:
If AI tends to recommend the most familiar package, what’s the strategy?
We can test whether clearer owned-content changes which pages get retrieved for buyer questions and whether that happens for the right questions.
So I ran a small before/after pilot in OppLab, Oppkey’s software for doing exactly this, using Flask vs FastAPI as the case study. This was based on real-world development and getting (repeated) recommendations from AI to use Flask when we eventually went with FastAPI.
In the test, if the content about Python frameworks is weak, then I add clearer pages about when to use FastAPI vs Flask, do the same buyer questions start finding those better pages?
Important limits:
- This was not a live ChatGPT experiment.
- We did not try to make FastAPI win every prompt.
- Success meant: API-fit questions find FastAPI guidance; traditional website questions do not get forced onto FastAPI.
How the test worked
- Before: Three short, vague, Flask-leaning pages. The kind of content that says “Flask is a common place to begin” without explaining when FastAPI fits.
- Ask 10 buyer questions Validation/OpenAPI, async APIs, Flask vs FastAPI, plus traditional site / tiny CRUD.
- Save an Evidence snapshot as Baseline.
- After: Add two better pages: a fair Flask-vs-FastAPI decision guide, and a short FastAPI tutorial on validation and OpenAPI.
- Ask the same 10 questions again.
- Save Evidence as After edit and compare.
What happened
Before, every question already found something but often the wrong, vague page.
After, 8 of 10 questions still had a #1 result, but that #1 page changed to the new guide or tutorial.
2 of 10 stayed the same, the “server-rendered website” and “tiny internal CRUD” questions. That’s a good outcome: better FastAPI content did not steamroll cases where Flask is reasonable.
| Result | Count | What it means |
|---|---|---|
| Shifted (#1 page changed) | 8 | Right content started winning |
| Unchanged | 2 | Traditional / simple apps stayed on generic pages |
| New hits / rank climbs | 0 | We weren’t climbing from “missing," we were replacing the winner |
In Evidence terms this shows up mostly as shifted, not improved. That’s a solidly good result. "Improved” means a new hit or a better rank number. Our questions were already hitting; the win was which page sat at #1.
Mean top score also moved up a bit (about 0.115 → 0.175) which is an indication that the new winners were a stronger match, not just a random reshuffle.
Examples of the shift
| Buyer question (short) | Before (#1) | After (#1) |
|---|---|---|
| REST API with typed validation + OpenAPI docs | Python Web Framework Basics | FastAPI validation/OpenAPI tutorial |
| When should I use FastAPI instead of Flask? | Vague “beginners” section | Flask vs FastAPI decision guide |
| Concurrent outbound HTTP calls | Framework basics | Tutorial’s async section |
| Traditional server-rendered site | Framework basics | Framework basics (unchanged) |
| Tiny internal CRUD admin | Getting started overview | Getting started overview (unchanged) |
“Shouldn’t we just publish more content?”
If models default to popular tools, “publish more” alone is fuzzy advice. A sharper loop looks like this:
- Write the buyer questions people actually ask, including the ones where your tech should and shouldn’t win.
- Score your current pages against those questions.
- Add content that teaches fit. When to choose you, when not to.
- Measure whether the right pages start winning those questions.
This is what OppLab is made for.
Straight Forward Summary
Better content didn’t make FastAPI appear from nowhere. It made the right page win for API questions, without stealing the traditional-website questions.
That’s not stunning, not magic, but it’s a valuable result to understand, and it’s exactly the kind of thing you want to measure before you bet a whole campaign on a rewrite.