# Methodology

This file is generated by pipeline/methodology.py from live values at export
time. Do not edit it by hand; edit the generator or the configuration it reads.

Generated 2026-10-19T08:15:59+00:00. Ruleset version 2026.10.1. Schema version 13.

## What the system does

A scheduled job polls the RSS feeds of every active outlet in
sources/outlets.yaml, stores every item before doing anything else, and
applies a loose multilingual keyword gate to the title and summary to decide
whether an item plausibly concerns China. The schedule asks for one run an
hour; GitHub Actions delays and drops scheduled runs under load, so the actual
interval varies and every run's real start and finish are recorded in the run
log. The gate's terms include every Chinese province and the largest cities,
because short wire and release items about one province often name no other
China term. An item whose title or summary already carries a state origin
signature passes the gate whatever its keywords say, so the gate never
discards the material the instrument exists to find. Outlets in Hong Kong,
Macau and Taiwan do not pass the gate on the name of their own territory, and
outlets in Singapore and Malaysia do not pass it on phrases that describe their
own Chinese communities ("Chinese New Year", "Malaysian Chinese Association"),
so local news is not counted as coverage of China.

Some busy outlets are also read through their Google News sitemap, which lists
every recent article with a title and no summary. For those sitemap items only,
the gate also reads a short list of title terms (the title_only block of
pipeline/keywords.yaml): Chinese places, state issuers, trade fairs and
initiatives, and some companies. Where a feed summary would say China, a
sitemap headline often names only the city, the fair or the company, so that
name is the only signal there is. Items from ordinary feeds, which carry a
summary, are never judged on this list. A company is kept on it only where,
among the stored titles it alone would admit, those labelled as China coverage
at least match those labelled not about China; most car, phone, appliance,
solar and drug makers fail that test and are left out. A match is recorded as
"title_only:<term>" among the item's gate terms. On stored PR Newswire titles
the list raised the gate's recall of state origin releases from 103 to 145 of
210, at a cost of about 8 to 25 extra article fetches a day.

The gate has its own version, 2026.10.1, separate from the ruleset: it
decides what enters the corpus, not how an article is labelled. Gate decisions
are recorded per item and never revisited, so a change to the gate applies only
to items discovered after it. Items an earlier gate rejected are not
re-examined, and rejected items are pruned after 3 days, except a
random 2 percent kept permanently for the gate audit. Re-running a
new gate over the rejections still stored measures only what is recoverable
from them; it cannot speak for items already pruned, or for feed entries that
rolled off a feed before any poll saw them, so a small recovery is never
evidence that the earlier gate lost nothing.
The gate last changed on 2026-10-12, to version 2026.10.1.

Where an outlet's press release, sponsored or partner section has been found
(sources/outlets.yaml, release_sections and section_feeds), that section is
polled as well, and its items skip the gate: RSS feeds usually carry editorial
coverage only, while paid placements sit in those sections. At export time 1386 of 1700 active outlets had been searched for such sections: 194 had one, 1084 had none, 11 blocked the search and 97 could not be reached. Where no section was searched, state origin placed through that route cannot be found, and the count per country is part of every result.

Items that pass the gate are fetched in full, honoring robots.txt, at most one
request per domain every three seconds, or less often where the site's
robots.txt sets a longer Crawl-delay (honored up to ten minutes; feed polling is
paced the same way), with an identifying user agent. Pages
that declare themselves not free in schema.org metadata, or that present a
paywall interstitial with a short body, are recorded as paywalled and never
retrieved by any other route. Paywalled articles are never classified, so they
are missing from every count, and from the share of China coverage, where an
unread article is in neither half. They do remain in the share of monitored
output denominator, which counts every item the outlets published whether or
not it could be read, so in a country with many paywalls that share is a lower
bound. A country where paywalls pass a third of retrieved articles carries a
warning on every figure the interface shows for it.

Fetched articles pass through a deterministic signature matcher
(pipeline/signatures.yaml) that detects state origin text by credit lines, datelines,
distribution stamps, sponsored placement disclosures and diplomatic bylines.
Any strong signature, or weak signatures from two different groups, gives
the state origin label with confidence 1.0 and no model call. One weak signature, or any
official sourcing trigger (a citation of Xinhua, CGTN, Global Times, China
Daily, CCTV or People's Daily; a quoted Ministry of Foreign Affairs
spokesperson, Chinese embassy or named official spokesperson; or a
"state media reported" construction in any registry language), routes the
article to a language model for the unchecked state sourcing versus independent
journalism judgement. An article with neither whose headline, feed summary or body
nearly repeats a state origin item another outlet ran from eleven days before it to eleven
days after it goes to the model as a possible copy of state text run without its
credit line (pipeline/lookalike.py), and the model is shown that item's headline and
outlet. Articles with no
signature, no trigger and no such match are labelled independent journalism by rules when
the body substantively concerns China (at least three distinct China terms,
or five occurrences, or a term in the headline with two occurrences),
otherwise not relevant. That threshold applies only to articles no signature
has claimed. Those residual labels carry confidence below 1.0 and are eligible
for the reliability study like every other label.

The gate in front of the model cannot see uncredited relay. Agency copy that a newsroom runs without a credit line names no state outlet, spokesperson or embassy, so it trips no signature and no trigger and, unless another outlet carried the same text as state origin, is labelled by rules, usually as independent journalism, without the model ever reading it; outlets that habitually run wire copy unattributed are where this is most common, and where the counts of state origin and unchecked state sourcing are most likely to be too low. The recall of the gate is measured by hand (pipeline/recall_audit.py): a seeded random sample of the articles the rules settled without a model call, stratified by language and oversampling outlets with a wire copy habit and articles whose headline or summary nearly repeats a state origin item already in the database, is coded by a person who does not see the rule label, and the share placed in state origin or unchecked state sourcing, weighted back to the population, is the gate's miss rate. The miss rate is not yet measured.

The model (claude-sonnet-5) sees the headline and body only. It never sees the
outlet name or the country, so it cannot learn that outlets in particular
countries tend to relay official sourcing. Its prompt contains the category
definitions verbatim and instructs it to answer independent journalism when
genuinely uncertain, because under-counting unchecked state sourcing is the safer error.
At most 4000 model calls are made a day, and the stage also
keeps to a budget of 110 dollars a calendar month: what is left of
the budget at the start of a day is spread over the days remaining in the month
and turned into a second cap on calls, and the lower cap binds. Spending is
estimated from recorded token usage at published prices; the estimate for
2026-10 so far is 46.61 dollars. When more articles are
waiting than the ceiling allows, the articles sent are a stratified random
draw: every country gets the same sampling fraction of its waiting articles, so
the countries whose articles arrive late in the UTC day are not the ones the
ceiling truncates. The eligible and drawn counts are recorded per country per
day, as a diagnostic. They are not a reweighting basis and no count here is
reweighted by them: the two numbers are recorded at different moments in a day,
and a draw that takes the same fraction from every country cannot repair a
comparison between countries anyway. Articles not drawn stay waiting for a later
day. Drawn records the calls actually made, so a country's figure can be zero on
a run that stopped at its time budget before working through its allocation;
that is a run ending early, not a country left out by design.

When the monthly budget is exhausted, waiting articles may be judged by the
same model with the identical prompt through an interactive session on the
owner's machine, at no API cost. The session is given the headline and body
only, never the outlet, the country or the URL, exactly as the API is. These
labels carry a distinct model version that names the session, are committed to
the repository and imported once by the runner, only for articles still
waiting and never over an existing label, and are subject to the same
reliability study rules as every other model label.
At export time 4557 current model labels had come in this way (claude-sonnet-5 (session): 4557).

The model stage first takes the articles whose text the session could not fetch on the owner's machine (refused, gone or paywalled), which only the API can judge from the runner's own copy, and then the other waiting articles, drawing within each group as described above. Since 8 October no article is held back for the session: the monthly budget is sized for the API to judge every article the rules cannot settle, and the session route is used only when the budget runs out.

The model prompt is dated 2026-10-12, and every model label records in its model version the date of the prompt it was made under. The prompt of 2026-10-12 added the factual-claim rule given under the categories below. When the prompt changes, current model labels made under an older one are judged again through the API: unchecked state sourcing labels first, since the factual-claim rule changes that figure most, then independent journalism labels that cite official Chinese sources with no independent confirmation, then state origin labels, then the rest, oldest first within each, with what the new articles leave of each run's calls and never more than 50 percent of the day's call cap, paced over the day. Each keeps its old label until the new one is stored, so no article is left without a label in between, and the timeline marks the days whose labels still carry an older prompt. At export time 9,243 current model labels did.

The instrument exists to find two kinds of article: state placements and
local pieces that carry Chinese official or state media claims without
checking them. Ordinary China coverage is only the denominator. Articles whose
official sourcing has been detected but whose verification judgement has not
yet run are reported separately as candidates, never folded into either label.
At export time 155 such candidates were waiting across 49
countries.

## The categories

State origin. Text authored or commissioned by a Chinese state entity and published without substantive editorial alteration.

Unchecked state sourcing. Written by the local outlet, and its central content is a factual claim attributed to a Chinese official or state source (an assertion of fact about events, numbers or actions), passed on without independent confirmation. A stated position, policy, warning, threat, opinion or denial is not a factual claim, and reporting one is independent journalism even without outside comment.

Independent journalism. The outlet's own reporting, including reporting that quotes Chinese officials but confirms, contextualizes or contests what they say, and reporting of a position, policy, warning, threat, opinion or denial attributed to a Chinese official or state source, with or without outside comment.

Not relevant. Does not concern China.

The critical distinction is between unchecked state sourcing and independent
journalism, and it is a judgement about verification, not about topic or tone.
An article highly critical of China that repeats an unverified Xinhua claim is
still unchecked state sourcing. An article sympathetic to China that independently
confirms every claim is independent journalism. Only a factual claim central
to the article counts: "Exports rose 8.7 percent in September, China's commerce
ministry said", reported as the figure with no other source, is unchecked state
sourcing, while "China's foreign ministry warned Manila it would take resolute
measures to defend its sovereignty", reported without outside comment, is
independent journalism, because it passes on a position and not a fact the
outlet failed to check. In the database and the data
files these four labels are stored as the codes A, B, C and not_relevant, in
that order, for compactness; the interface never shows the codes. Unchecked
state sourcing is the interface's name for what the codebook and the model
prompt call unverified relay; the prompt keeps that name, and the data files
keep the column name unverified_relay.

## How state origin arrived

Every state origin label stores its route: the mechanism named by the
signature that established it. A wire credit line or dateline, a press release
distribution stamp, a sponsored or partner disclosure, or a byline by a Chinese
diplomat. A state origin label the model gave to an article the lookalike
check sent, a near repeat of a state origin item another outlet ran with its
credit line, has the route uncredited copy: state text republished without
that credit, judged by the model. Any other state origin label the model gave
without a signature is unattributed. Separately, every article records its
arrival: an editorial feed,
or one of the outlet's press release, sponsored or partner sections. The two are
different variables, and both are exported per day and per country and on each
article, so penetration can be analysed by the routes available in a country
rather than by the country alone.

State origin labels by route, all time: wire credit line or dateline 1028; press release distribution stamp 366; sponsored or partner disclosure 149; signed by a chinese diplomat 17; uncredited copy of a credited state item 178; no signature; model judgement 2035.

## Current state

| Measure | Value |
| --- | --- |
| Outlets registered | 2132 |
| Outlets active | 1700 |
| Countries monitored | 235 |
| Countries recorded as coverage gaps | 17 |
| Articles discovered | 3596181 |
| Articles passing the relevance gate | 108207 |
| Articles classified | 87857 |
| Articles with a human review | 0 (0.0 percent of classified) |
| Share of gated articles that were paywalled | 10.5 percent |
| Countries flagged as not comparable because of paywalls | Canada, Switzerland, Costa Rica, Denmark, Finland, Hong Kong, Japan, Norway, Sweden, Uruguay |
| State origin, all time | 3773 |
| Unchecked state sourcing, all time | 1971 (provisional) |
| Independent journalism, all time | 42509 |
| Language model calls, all time | 9047 |
| Days on which more articles waited than that day's cap allowed | 2026-09-16 (3975 calls made), 2026-09-17 (no calls attempted; the monthly budget was spent), 2026-09-18 (no calls attempted; the monthly budget was spent), 2026-09-19 (no calls attempted; the monthly budget was spent), 2026-09-20 (no calls attempted; the monthly budget was spent), 2026-09-21 (no calls attempted; the monthly budget was spent), 2026-09-22 (no calls attempted; the monthly budget was spent), 2026-09-23 (no calls attempted; the monthly budget was spent), 2026-09-24 (no calls attempted; the monthly budget was spent), 2026-09-25 (no calls attempted; the monthly budget was spent), 2026-09-26 (no calls attempted; the monthly budget was spent), 2026-09-27 (no calls attempted; the monthly budget was spent), 2026-09-28 (no calls attempted; the monthly budget was spent), 2026-09-29 (no calls attempted; the monthly budget was spent), 2026-09-30 (no calls attempted; the monthly budget was spent), 2026-10-01 (110 calls made), 2026-10-02 (109 calls made), 2026-10-03 (108 calls made), 2026-10-04 (107 calls made), 2026-10-05 (105 calls made), 2026-10-13 (483 calls made), 2026-10-14 (531 calls made), 2026-10-15 (530 calls made), 2026-10-16 (529 calls made), 2026-10-17 (534 calls made) |
| Feed polls that returned a full window with nothing seen before | 25492 of 547694 polls; an estimated 69464 items missed over the 108174 polls estimated since 2026-10-13 |
| Relay collector on the owner's machine | 111 outlets collected only there, in 80 countries; 148 more polled there and by the hosted runner, in 81 countries; last pass 2026-10-19T07:19:01+00:00 (0.9 hours before export); 37 incomplete days |
| Current labels by ruleset version | 2026.09.2: 85, 2026.09.7: 17687, 2026.09.8: 8745, 2026.09.9: 32212, 2026.10.1: 29128 (reclassification under 2026.10.1 in progress) |
| Last successful run | 2026-10-19T08:15:58+00:00 |

## Reliability and what may be published

No hand coding study has reported yet. The owner and two research assistants code a fixed random sample drawn by pipeline/agreement.py: about 200 articles in each language with at least 300 model labelled articles, stratified by language and, within each language, by the model's current category, from articles with an extracted body, and seeded so the draw can be repeated. The stratification puts unchecked state sourcing, rare in most languages, into the sample in numbers that can test it, so the kappa describes that sample and not the proportions of the whole dataset. Coders work blind: the sheet carries headline, link, outlet and language, never the model's label, and data/reviews/CODING_GUIDE.md sets out how they decide. The current sample, human_sample_2026-10-12.csv, holds 1400 articles: en 200, es 200, ms 200, ko 200, ar 200, fr 200, pt 200. Until that study reports, nothing here is validated against human judgement.

Unchecked state sourcing counts are shown as provisional: they are one model's judgement and no hand coding has checked them yet. Every figure that contains them is marked provisional in the interface and carries relay_status provisional in the CSV export. No hand coding study has reported yet; the sample is drawn and being coded. The counts stay provisional until the owner and two research assistants have hand coded a fixed random sample, about 200 articles in each language with at least 300 model labelled articles, stratified by language and, within each language, by the model's current category, and Cohen's kappa between their codes and the model's on unchecked state sourcing versus independent journalism is at least 0.6, with at least 10 articles on each side, overall and in every language with enough items. If the study reports kappa below 0.6, overall or in a language with at least 20 items in that comparison, the counts are withheld there. A stability check, described below, never takes the provisional marking away.

Stability check. The last stability check (2026-10-09, 225 articles, kappa 0.62 on the unchecked sourcing versus independent journalism judgement) was run under an earlier version of the classifier's instructions, so it is kept as history and neither withholds nor supports any count. Enough articles have now been labelled under the current instructions (at least 200), and a new check runs when the daily model allowance permits, on 2026-10-20 at the earliest.

## Planted bylines

Since 8 October 2026 the pipeline also keeps a watchlist of fictitious journalist and expert personas, and of false front organisations and outlets, that published investigations have named as having placed or pitched articles in real news outlets (sources/personas.yaml, every entry citing its report). Every discovered item, including those the China gate rejected, has its author field compared with the list after names are normalised and initial-less forms are derived; items with a body are also checked for a listed name after a byline word at the top of the text and for a listed front organisation or domain anywhere in the text. A match is recorded as a flag with the detector, the persona or front, the investigation and the matching span, and the item is kept past the usual retention window. A flag means only that the byline or citation matches a name a published investigation has exposed; the tracker cannot prove a byline is fictitious, and the absence of a match proves nothing. Flags never enter the three categories and never change a count. A recurring bylines diagnostic exists on the command line only (python -m pipeline.planted recurring) and is never published, because it lists syndicated agency reporters. At export time the list held 14 personas and 11 fronts from 10 investigations (list version 2026.10.1), and 0 flags had been recorded in all, 0 of them on articles discovered in the last thirty days.

## New measures (provisional)

These measures are report only: none of them changes a label, a count, a denominator or the outlets monitored. Each is marked provisional until its own hand check passes, and withheld if the latest completed hand check fails.

**Alteration score (provisional).** The alteration score tests the second half of the state origin definition, that the text is reproduced without substantive editorial alteration. It sets each state origin copy beside the original state text it reproduces and measures how much of that text the outlet changed. It is report only: it never changes a label, a count or a denominator, and a copy with a high score keeps its label. At the last export 706 of 3773 state origin copies had an original (72 from a reference source, 634 an earlier monitored copy), and 692 had both texts available to score.

An original is looked for first among the reference sources, the English and other editions of Chinese state outlets read only as originals. A reference item qualifies when the copy's credit names its publisher, it is in the copy's language, it appeared from eleven days before the copy to one day after it, and its headline, feed summary or body fingerprint nearly repeats the copy's; when both texts are available at least 30 percent of the copy's words must line up with it. Failing that, the original is the earliest credited copy of the same item at another monitored outlet, in the same language. Translations are never matched across languages, so an Italian copy of Xinhua English has no reference original and is counted as a translation. Copies first seen in the last 14 days are paired again at every export.

Both texts are compared after credit lines, datelines and lines an outlet repeats on many of its pages are removed. The score is the share of the copy's words that are not found in the original, in runs of three words or more. Cuts are reported separately as the share of the original kept, and headline changes as a headline similarity. The provisional bands are verbatim (under 3 percent new words), light (under 15 percent), moderate (under 40 percent) and substantial (40 percent or more).

The bands measure wording, not meaning. A copy can keep every word and still change the sense by cutting a paragraph, and a copy can score as substantial because an outlet added its own background or a rebuttal. Two keyword cues, one for local context and one for counterpoint, flag added text for a reader to check; they are weak and never a finding on their own. Bodies are stored only where the tracker runs, so only copies whose two texts are both available are scored, and the coverage line says how many that is. Every copy in the substantial band is listed for review by hand, and a blind hand check of 20 pairs per band tests both the pairs and the bands.

**Republication speed (provisional).** Republication speed is the time between a state original and a state origin copy of it in a monitored outlet. It is measured only for copies paired with their original (see the alteration score), and only when both publication times are credible: readable, not after the item was discovered, and not before collection began. The time the tracker discovered an item is never used as a lag. Where the original is the state outlet's own page, collected as a reference source, the lag is absolute. Where no such page was found and the original is the first monitored copy, the lag is relative, behind the first monitored copy, and the two are never pooled. Lags fall into four buckets: the same hour, the same day (1 to 24 hours), 1 to 3 days, and later. A copy up to an hour ahead of its original counts as the same hour, since clocks differ by minutes. When either page gives only a date, the pair is bucketed by calendar date instead (same date, 1 to 3 days, later) and kept out of the medians. Copies more than an hour ahead of their original, and pairs without a credible time, are counted but kept outside every distribution. An outlet gets a median from 10 timed pairs; a route, country or source from 5. Wires are counted under WLD. At the last export, 71 absolute pairs fell inside the distributions.

The times are what the pages say, and they are not always what they seem. Many sites give the time a page was last updated as its publication time: on the copy that makes it look slower than it was, and on the original it can put a copy ahead of its source. Some state listings label their time zones wrongly: a Beijing time written as UTC reads eight hours late. Times are corrected per source when they are collected, and the export checks every reference source for this by comparing publication with discovery: the median gap, the share published after it was discovered, and the share that give only a date. Pages that give only a date can say no more than the calendar day. State text also reaches outlets by channels the tracker cannot see, such as wire terminals, email and agreements, so a lag is the time between two public pages, not the time it took the text to travel. Coverage is partial: only copies paired with an original count, the reference sources reach back only a few days, and most copies have no original found. The hand check compares the bucket of 60 reference pairs, read by a person from the times shown on both pages, with the measure's bucket, and passes at 85 percent agreement.

**Doctrine terms (provisional).** The doctrine term tracker counts how often China coverage uses the Chinese state's own set phrases, such as community with a shared future, true multilateralism, Chinese modernization and the one-China principle. There are 13 families of terms. Each family is matched in English and in the renderings Chinese state media and the Chinese foreign ministry publish in the outlet's language; a language is left out of a family when no such rendering was found, and every language is also matched on the English terms. The terms are read in the headline, the feed summary and the whole body, and every occurrence is counted. A match is a keyword match, not a reading of the article: it says the phrase is there, not who used it or whether the article endorses it.

Some matches are excluded by rule. A bare win-win is ordinary English and never counts, only win-win cooperation; hegemonism counts but the generic hegemony does not; the one-China principle is kept apart from the one China policy other governments state. An acronym counts only when a full term of the same family appears in the same article, and GSI only beside Global Security Initiative itself, so BRI in Indonesian football or banking news is not the Belt and Road. Belt and Road and Global South are common well beyond Chinese state usage, so they are shown muted and left out of the share of articles using the vocabulary.

Rates are per 1,000 China articles of each kind (state origin, unchecked state sourcing, independent journalism) on the same days, counting only articles already tagged under the current term list. Where a body has not been read the headline and summary alone are matched, which finds fewer phrases; the article is tagged again once its body arrives. A hand check codes up to 200 matches, spread over families and languages, each shown with about 30 words either side. A family is checked at a precision of at least 0.9 on at least 15 coded matches and withheld below 0.8 once 5 are coded, and a withheld family shows no figure; the measure as a whole is checked when the matches of the families still shown reach 0.9 together.

**Unusual days (provisional).** Unusual days are days on which a country, or the worldwide press release wires, carried far more state origin placements than usual. Each complete day is compared with the 28 complete days before it, counting only the days on which the country's outlets were collected at all, and at least 14 of them are needed. The expected count is the mean of those days after the busiest tenth of them is set aside, and never less than 0.2. A day is flagged when it has at least 5 placements, at least three times the expected count, and a chance under 1 in 1,000 of reaching that count if placements arrived at random at the expected rate (a Poisson model). The day the export runs is partial and is never tested. China is not monitored and has no series. Over the 7,113 country days tested so far, the Poisson model expects about 0.1 of them to be flagged by chance alone, a false flag rate of about 0.2 in 10,000 tests; 20 days were flagged.

Each flagged day lists the outlets and underlying items behind it, and the checks that could make it an artefact of the instrument rather than a surge of state text: a newly added outlet, a backfill read, a collector recovering from an outage, an incomplete day for the collector on the owner's machine, a ruleset or relevance gate change, saturated feeds on the baseline days, the daily model cap, and a surge in everything discovered rather than in state text alone. A day with any of these is shown muted with its reason, as one to check, and a day with none is shown as clean. The repeats in the discovery counts before they were fixed are noted too, but they touch only the rate per discovered item, not the count. Unusual days are report only: they change no count, and a hand check of up to 30 flagged days, each judged a real surge or an artefact, has to find at least 80 percent real before the marking provisional comes off.

## What the numbers mean

The default measure is state origin only, because it is the one label decided
without the model. There is no combined state-linked measure: adding the
candidates still waiting for the verification judgement to judged counts would
mix a measured quantity with an unmeasured pile whose composition varies by
country. The waiting candidates are shown only as their own figure, in the
country panel and the CSV exports, and are not a finding.

Every figure counts an item on the UTC day it was discovered, the only date the instrument observes. The exception is an item brought in by a backfill read (the AAP release reader's catch-up reads, a feed's first poll, and its first poll after more than 3 days without one) and discovered more than 36 hours after it was published, which counts on its publication day if that is on or after 2026-09-03, in the counts and in the share of monitored output denominator alike, because a backfill would otherwise put weeks of items on one day. A backfilled item on a day before a gate change marked on the timeline was admitted by the gate in force when it was read (version 2026.10.1 at this export), not by the earlier gate that applied to the other items of that day.

Raw counts mostly measure how many outlets the registry happens to track in a
country. Articles per monitored outlet corrects for registry density. The share
of monitored output divides by every item the country's monitored outlets put
in their feeds in the window, whether or not it concerns China;
the largest outlets are the 30 best audience ranks recorded in
sources/outlets.yaml, and 214 countries carry ranks so far and every other country uses all of its active outlets. A share of published items is not
shown for a country with fewer than 5 outlets behind it, where it
would describe a handful of feeds rather than a country. No per capita figure
is shown, because population is a poor denominator for what outlets publish.
The resident population of each country is still exported, in the population
column of the CSV of the current view, and no figure is divided by it.

A syndicated wire item placed in thirty outlets and thirty separate items are
different phenomena. Near-duplicate titles within a country are linked as one
underlying item; the interface shows both the number of placements and the
number of underlying items. An underlying item counts once in a window
however many of its days it was placed on, and once in a world figure however
many countries carried it. It counts under each label it carries, so an item
with a state origin placement and an unchecked state sourcing placement is in
both counts.

Titles are the first pass. An outlet often rewrites a wire headline, so a second pass compares the fetched text. Every body is cut into overlapping 5 word shingles (5 character shingles in scripts written without spaces) and summarised as a 64 value MinHash signature. Two articles from different outlets, in any countries, discovered within 7 days of each other, are linked as one underlying item when their signatures estimate that at least 50 percent of their shingles are shared. Pairs from the same outlet are left to title matching, and a body that matches another article of its own outlet under a different headline is treated as page template text and never linked. Measured on 6,855 stored bodies on 2026-10-12, the body pass linked 124 pairs of articles that title matching had left apart, 104 of them across countries, and none of a hand checked sample of 30 of those pairs joined two different items (one joined two successive versions of one wire story, and one a newspaper's rewrite of a news agency's copy); below the threshold, pairs began to join different versions or different stories about one event. Translations are not matched: a translated copy shares almost no words with its original, and matching it would need machine translation, which the tracker does not do. Body links change no category count, and the model's label is copied only within a title group, never along a body link. Body links can only lower the number of underlying items, and only where the linked placements fall in the same figure and the same category: in a country's figure when both placements are in that country, and in a world figure wherever they are. A link across countries marks both placements as syndicated and leaves each country's own figure as it was. Underlying items are counted once per window, so a story placed on several days of a window is one item, not one per day.

Global press release distribution services (us_businesswire, us_globenewswire, us_prnewswire) are not national
newsrooms: their releases are written by their issuers and pushed to
subscribers everywhere at once. They are polled and classified like any
outlet, and they are counted as a stratum of their own, "Worldwide
distribution", under the code WLD in the data files. The stratum appears in
the ranked list, the country search and the CSV exports beside the countries,
and nowhere on the map, because it has no territory to draw. It is never
attributed to the United States, where the services are registered, and never
added into any individual country's numbers: a release reaches audiences in
every monitored country, and adding it to each would count it once per
country. World totals include it. Its share of monitored output divides by
everything the wires themselves published. At export time the stratum held 1095 gated articles: 349 state origin, 2 unchecked state sourcing, 37 independent journalism, 692 not relevant and 10 awaiting the verification judgement. A Distribution toggle beside the Measure control shows
the two apart: Both is the whole dataset, Country outlets leaves the wires out
of every figure that sums across countries, and Worldwide wires shows the
stratum alone, with the map left unfilled because a release reaches every
country at once; the count of outlets monitored and the leaning chart count
newsrooms only and never include the wires in any mode.

The color scale is fixed for a measure and window across every date on the
timeline: the darkest step starts at the 95th percentile of the values of all
dates, not of the day shown, so a color means the same number on every date and
the countries above the cap sit in an overflow step. The cap holds only until
the next data update. Each update, about once a day, adds the newest days and
recomputes the cap over the timeline as it then stands, so a color can stand
for a different number from one week to the next. The theme counter's cap
works the same way.

Absence of data and absence of content are different things. A country with no
monitored outlets is drawn with a hatch pattern. A country whose outlets all
publish in a language without a keyword list is drawn as unreadable, because
the gate can only match international and English terms there. A country with
monitored outlets and no detections is drawn flat. Only detections above zero
use the color scale. A country whose feeds are mostly failing, whose articles
are mostly paywalled, whose feeds lose items between polls, or whose relayed
outlets were not collected carries a warning that travels with every figure
for it, in the tooltip, the ranked list, the country panel and the CSV export.

Feeds carry only their most recent entries. When a poll returns a full window
with nothing seen before, items published since the previous successful poll
were probably missed; a skipped run costs a busy media market more than a quiet
one. Every poll records whether that happened, and the estimated number of
missed items is the feed's own publishing rate times the gap. The rate is taken
over at least two hours, or over the feed's median window span in the last
three days when that is shorter (never under a quarter of an hour), so a window
stamped within minutes cannot imply hundreds of items an hour. One poll's
estimate is at most twice the new items the feed delivered per day over the
previous fourteen days, scaled up for a gap longer than a day. A feed polled at
least three times a day that delivers no more than two and a half windows a
day publishes in batches: a poll of it whose entries are stamped within an
hour of one another counts only the batches that fell wholly between it and the
previous successful poll, none when that poll came less than one batch
interval earlier. Until 2026-10-12 the rate was taken over the window alone
and five such batch feeds were booked about 15,000 phantom misses. Each day's
poll record names the estimator that made it, and the older estimates cannot be
made again because single poll times were not kept, so the published estimate
counts only the polls made by the current estimator, from the day it was first
used, while polls and full windows count every poll on record. The relay
collector on the owner's machine records every hourly pass; a day on which it
ran in fewer than 12 hours is marked incomplete for the countries
whose outlets only it collects, so a sleeping laptop never reads as a decline.
The outlets it shares with the hosted runner are not marked and carry no
incomplete warning when it falls silent: the runner still polls their feeds on
its hourly schedule and fetches their pages, so they lose only what a skipped
or late runner poll loses, which the full window count above records. Their
countries carry a milder warning while the relay is silent.

Themes. Every article that concerns China is tagged with each theme whose
terms appear in its headline, its feed summary or anywhere in its body
(pipeline/themes.yaml, themes version 2026.09.4): diplomacy and summits,
trade and economy, Belt and Road and investment, technology, energy and
electric vehicles, military and security, Taiwan and Hong Kong, human rights,
Chinese domestic politics, culture and tourism, and disasters and health.
Until themes version 2026.09.4 only the first 800 characters of the body were
read, which favored whatever an article names early.
Themes versions, each with the day it took effect: 2026.09.3 from 2026-09-14 (theme counter introduced; terms read in the title, the feed summary and the first 800 characters of the body); 2026.09.4 from 2026-09-15 (terms read anywhere in the body instead of the first 800 characters; every article re-tagged). Every export re-tags each relevant article whose tags were made under another version, so every article carries the tags of the current version and the published series has no break at a version change; the marks on the theme timeline record when the method changed, and a theme count read or downloaded before a mark was made under the earlier version. An article can carry
several themes, so theme counts do not add up to article counts; one with none
is Other. Theme terms exist for ar, de, en, es, fa, fr, hi, id, it, ja, ko, ms, nl, pl, pt, ro, ru, th, tr, vi; articles in other languages
are matched on the English terms only. The tags are keyword matches, not a
reading of the article, and should be checked by hand before a theme count is
cited.

## Reproducibility

Every classification row records the ruleset version and, for model labels,
the model version. When the signature list changes, the ruleset version is
bumped in pipeline/config.py, the change is recorded in CHANGELOG.md, and every
earlier article is reclassified under the new ruleset during the hourly runs,
not only new ones. Until that pass finishes, the timeline marks each ruleset
change and each day that still carries labels from an older version, so a
discontinuity is never read as a trend. Earlier rows are kept and marked not
current, so a chart made from the database at any date can be reproduced.
Human review labels are stored in a separate table and never overwrite
machine labels.

The database itself is not committed to git, because a growing SQLite file
exceeds GitHub's file size limit within weeks. It travels between runs
through the GitHub Actions cache, together with the extracted article
bodies, and is snapshotted daily, compressed, as a release asset. Raw HTML
is kept for the duration of a run and on any machine that runs the fetch
stage locally; the page chrome labels the sponsored placement patterns need
are extracted at fetch time and stored with the article, so they survive
without the HTML. The permanent record in git is data/export, one JSON line per
gated article per month with its current classification, route and arrival,
plus the site data under docs/data. One deliberate exception to the never-delete
rule: items the relevance gate rejected are pruned after 3 days. They
carry no classification. A random 2 percent of them is kept for
audit, permanently: the choice is made by a hash of the item's address, so it is
reproducible and does not depend on outlet, country or language, and each kept
item keeps its title, summary, address, outlet, country, language and the gate
terms it was judged on. pipeline/gate_audit.py draws its hand-checked sample from
that set, stratified by language, over any period, so the gate's recall can be
measured for months already past and not only for the last 3 days.
The address of every pruned item is remembered, for 120 days after a
feed last listed it, or 14 days after its discovery when no feed
lists it again, so a feed that keeps an item listed for weeks does not bring it
in, and count it among the items screened, a second time. Before
2026-10-14 it did: after every prune such an item came back as new. From
2026-10-01 to 2026-10-11, 8.2 percent of the items screened
were such repeats, from almost nothing in Serbia and Croatia to half in Sweden
and nearly three quarters in the Cayman Islands. Since collection began at
least 4.1 percent were (123,682 items the stored
database snapshots show coming back); applying each outlet's share in those
eleven days to the whole period puts it near 8 percent of
the running total of items screened, and the site says so under that total.
The counts before 2026-10-14 are left as they were: the addresses that
would show which items came back were pruned with the items, so they cannot be
corrected item by item, and a correction of only the days the snapshots cover
would put a step into every series. Each country's share of its own output (%
of all its news) divides by these counts, so over days before
2026-10-14 it reads low by that country's share of repeats, which is
uneven: about 16 percent in Britain, 18 in Vietnam, 28 in Uruguay, 47 in
Kuwait, 50 in Sweden and 72 in the Cayman Islands. The map's key says so for
any window that reaches back before 2026-10-14.