A repeated question can produce different answers, so one missing recommendation does not establish a visibility decline. Record the product surface, context, search use and time, then compare repeated observations under a consistent protocol before changing your content.
Community questions informed this guide. Linked provider documentation supports product-specific facts; the worked example is illustrative.
Separate a changed test from a changed result
First check whether the task stayed the same. A follow-up may add a budget, location or product preference. A search-enabled answer can use different evidence from a conversation without observed search. ChatGPT search documentation also describes location and context-sensitive behavior. These are reasons to record conditions, not a diagnosis of every individual difference.
Do not present a fresh conversation as a representative sample of all customers. Its value is that it gives you a repeatable starting condition you can document. If you cannot observe a setting or product change, mark it unknown.
Repeat a fixed panel without choosing favorable runs
Decide the questions, repeat budget and review window before inspecting the results. Preserve every reviewable answer and record collection failures separately. Additional runs of one prompt describe variability for that prompt; they do not automatically add more distinct buyer intents.
Choose a repeat budget your team can maintain and review. There is no universal number in this guide that makes an AI visibility result statistically representative. If collection conditions are strongly dependent or the prompt panel is selected by the brand, a large run count does not remove that bias.
Compare rates and response evidence together
Use a fixed definition of recommendation. Naming a brand in a warning is not automatically a recommendation, and linking to its documentation is not automatically an endorsement. Inspect the statements and sources behind any change.
When a comparable panel declines, investigate whether the change is concentrated in one question, one surface or a particular source claim. Preserve the old and new evidence before attributing a movement to a page edit.
| Observed pattern | Appropriate next step |
|---|---|
| One live demo omits the brand | Record the result and compare with the existing panel. |
| An engine or search mode changed | Label a break in the series; establish a new baseline. |
| Many comparable answers repeat an outdated fact | Verify and correct the underlying public source. |
| Collection failures increased | Resolve collection coverage before interpreting the score. |
What to check
- Keep the counting rule and panel stable.
- Report repeat observations and collection coverage.
- Investigate the changed claim rather than promising a stable rank.
A practical example
Illustrative example: 3 of 10 valid answers recommend a brand in one window and 4 of 10 in the next. Report 30% and 40% for that small observed panel. The difference is not, by itself, proof that a content edit improved market-wide visibility.
Sources and further reading
- OpenAI: ChatGPT Search and citations
- Arrow: AI citation tracking methodology
- Reddit discussion: inconsistent AI search visibility checks — anecdotal question signal
These steps are Arrow AI's implementation guidance. The example is illustrative; it is not a measured customer result. Publication and source access do not guarantee an AI recommendation.
Apply this to your website
Identify the source gaps and decide what to improve with Arrow's free audit. Review what Arrow GEO measures before choosing a reporting scope.
