When VisibilityOS by PodPitch compares brand presence in AI answers, the questions and conditions matter. If you’re comparing two groups around a website update, check how each group was moving before the change.
Here’s a fictional example. Each group contains 20 fixed questions, and you ran both groups on three dates before updating a page.
Questions about the updated page. The brand appeared in 5, then 6, then 7 of 20 answers. Its rate rose from 25% to 35%.
Comparison questions. The brand appeared in 9, then 8, then 7 of 20 answers. Its rate fell from 45% to 35%.
Both groups ended at 35% on the last pre-update date. The rates had already been moving in opposite directions. A later gap would need to be read in light of those earlier movements.
Keep a stable set of questions for each group. Save the counts, dates, and the rule you used to decide whether a brand was present.
Match the model, search settings, language, and market across rounds. Record any model release or setting change that affected one group.
Compare changes across more than one earlier round. Check whether missing answers or a change to the question mix affected either group.
If the earlier trends diverged, show each group’s observed change and the earlier trend. If you want to estimate an update’s effect, document why the groups would plausibly have followed similar trends without the update.
Use the VisibilityOS reporting guide for sample wording, the comparison calculator for the arithmetic, and the World Bank guide for the assumptions behind comparing changes.