A reproducible evidence log is a dated, public record in which every observation names the exact URL checked, the exact date and time it was checked, the intervention that preceded it, what was held constant, and every confounder you could not remove. Its test is simple: another researcher should be able to repeat your steps from your log alone and get either the same observation or a stated reason why not. If they cannot, it is a diary, not evidence.

Use the experiment designer to draft the protocol, then keep the observation record separate and public so that the protocol cannot quietly be rewritten after the numbers arrive.

What goes in each entry

Each dated line should be short enough to read and specific enough to check. A workable minimum set:

  • Protocol line. A short identifier, the research question, the host page URL, the comparison or control URL, and the observation window with start and end dates.
  • Intervention line. What changed, in plain language, and the date it went live. If nothing changed, say so explicitly, because a control period is also evidence.
  • Observation line. The metric you actually looked at, the value you saw, and the method you saw it by (which report, which query, which tool, which date).
  • Evidence URL. A stable link to the primary source or to the page or report the observation came from, with a retrieval date.
  • Confounder note. Anything else that plausibly moved at the same time, including site-wide changes, policy announcements, or seasonal demand.

Hypothetical example: a researcher records that on a stated date a third-party article page on a host site was moved under the host's editorial navigation and given an author and a conflict-of-interest note. The observation window is thirty days. The comparison is a sibling article left unchanged. The log line names both URLs, the date, and notes as a confounder that a broad core update was announced in the same month. That last note is not a caveat to be trimmed later; it is the entry's honesty.

Separate the inputs you control from the system

A log is only interpretable if it distinguishes what you changed from what the platform changed. Bing states its guidelines describe how it discovers, crawls, indexes, evaluates and surfaces content, and that failure to follow them may result in reduced visibility, reduced eligibility for grounding experiences, or delisting from the index (see Bing Webmaster Guidelines). Those consequences are described as possible, not certain, which is exactly why a single observation window cannot prove causation.

Google's March 2024 announcement describes a core update involving changes to multiple core systems whose rollout could take up to a month, and warns that rankings may fluctuate more than usual while systems update (see the Google Search Central blog post). Log the announcement date in your confounder column. Do not log it as a result.

A related trap: Google states that some third-party services provide reputation or authority scores for sites and that these do not correspond to any Google signals nor come from Google. If your log uses a third-party score as a column, label it as a third-party score, not as a platform signal, and never present it as a host score you measured.

Record what you did not measure

The part most logs omit is the negative record: which pages you did not check, which queries you did not test, which comparison you could not build. A useful public log names at least one thing it cannot answer. For example: this log does not establish that a change in ranking was caused by the intervention, because it has no impression data for the comparison URL before the window opened.

Bing's guidance supports measuring beyond clicks, noting that a decline in clicks does not always indicate a loss of visibility and recommending monitoring of impressions, indexing status, and grounding and citation eligibility. A log that carries only a click figure has already discarded the metric you would need to interpret it.

Keep the steps checkable

For each observation, the log should let a reader answer four questions without asking you: which URL exactly, which date, which method, and which source. Where the method involves a reporting interface, name the report and the query string rather than the conclusion. Where a source is cited, link the primary page. Google's spam policy pages describe illustrative factors for judging site reputation abuse, such as how content is presented relative to the host domain, differences in content quality, authorship indications, and whether identical or near-identical content appears on multiple sites, while stating that no single factor is necessary or sufficient and the list is not exhaustive (see Google's spam policies). An evidence log can record whether a factor is present or absent; it cannot convert that observation into a verdict.

For sitemap and notification changes, log the date and the mechanism. Bing describes IndexNow as a way to notify search engines about content changes as they happen, with incremental notification suggested for frequently changing content (see the IndexNow post). Those incremental thresholds are the author's suggested examples, not a published ranking rule, so log the threshold you chose and not a claim about its effect. Bing's Top Insights announcement recommends sitemap refreshes at least daily and lists example recommendations, but the figures and URLs shown in that announcement are demonstration content, not real measurements (see the Top Insights post). Never copy its sample counts into your log as benchmarks.

Redact, then publish

A public log does not require publishing everything. If a host page is confidential, publish the protocol line with a redacted identifier and keep the full URL in a private appendix referenced by a log entry, while still publishing the comparison, the window, the intervention description and the confounders. What must not be redacted is the reasoning chain, because the reasoning is what makes the entry reproducible.

Practical limitations to state plainly in the log: ranking systems are not stable instruments, observation windows are short relative to update rollouts, hosting a page is not a controlled experiment, and no public tool can tell you which signal moved. State also that absence of an observed change in a short window is not evidence of no effect.

If you are still choosing what to compare, read Design a comparison you can interpret first; if you are unsure which stage you measured, read Separate discovery, indexing and ranking. The about page describes what this notebook does and does not claim. Start a protocol in the study designer, record every dated observation against it, and publish the log even when the answer is that you found nothing.