A comparison you can interpret is one where a reader can see, before any result arrives, what was changed, what was held constant, what was measured, over what window, and what would count as a different outcome. If any of those five is missing at the start, the end of the study will not tell you anything. The fix is cheap: write the protocol first, in the experiment designer, before you touch a page.

This guide sets out a workable design routine. It does not report findings, because none have been measured here. Every example below is labelled hypothetical, and the only factual claims are the ones traced to the cited primary sources.

Start from a question that names its stage

Many comparisons fail because the question smuggles in a stage. "Did the host profile improve?" is unanswerable. "After URL consolidation, did the URL move from eligible-but-not-selected to selected for grounding-style results?" is answerable because it names a stage.

Bing's guidelines make this separation practical rather than pedantic. Crawl access is controlled by robots.txt, while indexing of a URL is controlled by directives such as NOINDEX: "robots.txt controls crawl access, not indexing" (Bing Webmaster Guidelines). A design that changes crawl behaviour and then reads a ranking metric as though crawl were the variable has collapsed two stages into one. Write the stage into the question, and see the companion note on separating discovery, indexing and ranking.

Fix the observation window before you begin

Decide how long you will watch, and why that length. Write both into the protocol. A window chosen after seeing a wobble is not a window; it is a search for the moment that suits your story.

Google's March 2024 announcement is directly relevant to timing: it describes a core update whose complex rollout "may take up to a month", with likely fluctuations as different systems update and reinforce one another (Google Search Central blog). If a study's window sits inside a period of acknowledged volatility, any observed movement is entangled with it. Record the window and the platform context together, so a later reader can see the overlap.

Hypothetical example: a researcher planning a four-week observation on a host site checks the release notes and discovers the study would straddle a announced core update. They either extend the window or wait. Either choice is defensible; not recording the overlap is not.

Choose a control you can defend in one sentence

A comparison needs something that did not receive the intervention. Three common choices, each with a cost:

  • Same host, different section. Close on site-level context, but sections differ in age, linking and oversight.
  • Same content type, different host. Close on format, but host histories differ in ways you cannot observe.
  • Before and after on the same URL. Controls for everything except time — which is precisely the thing that also drives platform change.

None is clean. The point is to state which confounders you accepted and why. Google describes its site reputation abuse evaluation as considering a set of factors — how content is presented relative to the host domain, quality differences from the main domain, authorship indications, and whether identical or near-identical content appears on multiple sites — while stating that no single factor is necessary or sufficient and the list is not exhaustive (Google spam policies). If the policy assessment is multi-factor and non-deterministic, a single-variable design cannot isolate it. Say so.

Hypothetical example: a study compares a hosted third-party section against the host's own editorial section on the same topic. The protocol records that editorial oversight differs by design, and therefore any difference cannot be attributed to hosting alone.

Describe the intervention in terms someone else could reproduce

"Improved the content" is not an intervention. An intervention is a dated, itemised list: which URLs, what changed, what did not. If a page gained an author byline and a visible statement of editorial responsibility, write that. If content was consolidated or redirected, write the redirect type. Bing states that 301 redirects are for permanent changes and that duplicate URLs dilute signals and reduce confidence in selecting a URL for grounding results or citations (Bing Webmaster Guidelines). A protocol that says "cleaned up URLs" cannot be checked against that guidance; one that lists each redirect can.

If notification is part of the intervention, record it. IndexNow is described in a Microsoft Bing blog post as a capability for notifying search engines about content changes as they happen, and the post suggests incremental notification thresholds for very frequently changing pages such as reviews and engagement counts (Bing blog on IndexNow). Those thresholds are the author's illustrative examples, not a published ranking rule. If you adopt them, cite them as a choice, not as a requirement.

Name the confounders and the metric before the data

Confounders to declare in writing, at minimum: concurrent platform updates; seasonality of the query; changes to the host's own navigation or internal linking; duplicate or near-duplicate content elsewhere; and any editorial or commercial change made for unrelated reasons. Each one you cannot control should appear in the protocol as a stated limitation.

Then choose the metric. Bing explicitly warns against reading clicks alone: a decline in clicks does not always indicate a loss of visibility, and it advises monitoring impressions, indexing status and grounding or citation eligibility (Bing Webmaster Guidelines). A design whose only outcome measure is clicks has pre-committed to an ambiguous result.

Hypothetical example: a protocol records impressions, indexed-URL counts and a manual check of eligibility, alongside clicks, and states in advance that a click decline with stable impressions will be reported as ambiguous, not as a loss.

Record evidence links, not impressions

Every claim in the write-up needs a URL and a retrieval date. Platform documentation changes; a claim without a date is not reproducible. The evidence log guide covers the logging discipline; the design point here is that source links belong in the protocol, not gathered afterwards to support a conclusion.

Be equally careful about third-party numbers. Google states that some third-party services provide "reputation" or "authority" scores for sites and that these do not correspond to any of Google's own signals nor come from Google (Google Search Central blog). Such a score is not a variable you can validate against platform behaviour, so it cannot be a primary outcome in a defensible design.

Export and register before you start

Use the study designer to capture the question, host and page, observation window, comparison or control, intervention description, confounders and evidence URL, then export the protocol as JSON. The export is the pre-registration: a timestamped file that fixes your design before data exists. Store it, and treat later edits as amendments with their own dates rather than silent revisions.

Practice limits worth stating plainly

  • Single-site studies cannot estimate platform-wide effects; they describe one URL set over one window.
  • Policy documents describe evaluation criteria, not effect sizes, and Google states no single factor is necessary or sufficient.
  • Bing notes it provides no ranking-weight data and that SEO does not guarantee rankings or traffic, and grounding does not guarantee citations.
  • Any number shown in a product demo, such as sample pages or illustrative totals, is demonstration content, not a benchmark (Bing Top Insights announcement).

Design well and the result may still be inconclusive; that is a finding, not a failure. Design badly and the result will be a story about your own choices. For context on why the label invites both, read what the parasite SEO label hides.