What News Sitemaps Actually Look Like
Samvad, the news tool I wrote about earlier this year, answered questions about Indian news coverage by citing real articles. I’ve written about its agents and its retrieval. I never wrote about where the articles came from.
They came from a small Go service I called the harvester. Every fifteen minutes it read the news sitemaps of Indian outlets, 26 of them by the end, worked out which articles it hadn’t seen before, and pushed each one onto a queue as a JSON event. At its busiest that was six to eight thousand articles a day.
Samvad is archived now. The harvester is still around. I rewrote it this month and it’s open source. A good part of it is there because a sitemap somewhere did something the spec says it shouldn’t, and that’s what this post is about.
flowchart TD
subgraph IN["In: publishers"]
S1["Open news sitemaps<br/>(most outlets)"]
S2["Akamai-protected sitemaps<br/>(a few outlets)"]
end
S2 --> PH["Home Android phone<br/>(residential IP, over Tailscale)"]
S1 -->|"direct"| H["Harvester<br/>(Go, on a GCP VM)<br/>fetch, dedupe,<br/>read page head"]
PH -->|"proxied"| H
subgraph OUT["Out: sinks"]
PS["GCP Pub/Sub"]
X["Webhook, SQS, SNS, log<br/>(supported, unused)"]
end
H --> PS
H -.-> X
subgraph SV["Samvad"]
W["Ingest worker<br/>(dedupe on URL hash)"] --> DB[("Postgres")]
DB <--> EW["Embed worker"]
DB --> A["Ask and Coverage Report"]
DB --> PU["India News Pulse"]
end
PS --> WHow it was wired while Samvad ran. Most outlets were fetched directly, the blocked ones went through a phone at home (more on that below), and everything left as events on one Pub/Sub topic.
Why news sitemaps over RSS
A news sitemap is an XML file a publisher puts up for search engines, usually
advertised from robots.txt.
Google’s guidelines
for them are short: list only articles from the last two days, put at most
1,000 in a file, and give every entry a title and a publication date.
I went with sitemaps over RSS for licensing reasons. Honestly, feeds are easier to parse. But plenty of publishers restrict their RSS to personal, non-commercial use in their terms. A sitemap is published for crawlers to read, which is a far better fit for what the harvester does.
The pipeline that falls out of that is small:
flowchart TD
A["Fetch sitemap"] --> B{"Root element?"}
B -->|"urlset"| C["Read url entries<br/>(capped at 5000)"]
B -->|"sitemapindex"| D["Fetch each child<br/>(depth 4, 200 children)"]
D --> B
B -->|"html, rss, anything else"| E["Error: not a sitemap"]
C --> F["Canonical URL,<br/>then SHA-256 id"]
F --> G{"Seen in last 120h?"}
G -->|"yes"| H["Skip"]
G -->|"no"| I["Read page head:<br/>description, image"]
I --> J["Deliver to every sink"]
J -->|"all accepted"| K["Mark seen"]
J -->|"any sink failed"| L["Leave unmarked,<br/>next crawl retries"]One pass per source. All the sources run side by side, and one that fails doesn’t stop the rest.
A sitemap entry gives you a URL, a title and a date. There’s no description, so
for each new article the harvester reads the page’s <head> for its Open Graph
tags and stops there. It never fetches article bodies.
Where Google’s spec and real sitemaps disagree
On 6 September I recorded the live sitemaps of 23 of those outlets as test fixtures, keeping the first 200 entries of each. That’s 3,728 URLs, and most of the numbers below come from those files.
The two-day window is a suggestion. Ten of the 23 files had entries older
than 48 hours, and that’s counting only the first 200 of each. A regional daily
had 69 of its 200 past the line. A digital-native outlet had 154, the oldest
going back nine days. And a national daily listed an article from May 2023 in
its recent-news sitemap, with a lastmod stamped the morning I captured it. The
page had been touched, so back into the sitemap it went.
That matters because of how the harvester remembers what it has already sent. Every delivered article’s ID goes into a small bbolt database with an expiry of 120 hours. Google says two days, so I figured no article, even on a small site, would stay in a news sitemap longer than five. The note I left says “5 days > the 2-day sitemap window.” That holds right up until a source keeps nine days. In the nine-day outlet’s first 200 entries, 77 were older than 120 hours. Once an ID expires, the next crawl reads the article as new and sends it again.
What caught it was the other end. Samvad’s ingest worker checked its own SHA-256
hash of every URL before writing anything, so the second copy got dropped and
nobody ever saw it. I only worked this out while counting for this post. The
harvester’s contract already says delivery is at-least-once and consumers
should deduplicate on article.id. I just hadn’t pictured this case when I
wrote that sentence.
A publish date is a claim. Seventeen of the files write dates with a
+05:30 offset and six use UTC, all of it valid RFC 3339. The trouble came
from entries with no date, or a date in some other shape. The first version of
the harvester parsed exactly one layout, RFC 3339, and returned Go’s zero time
for anything else. Serialised, that’s
0001-01-01T00:00:00Z, which is a perfectly valid timestamp. So Samvad didn’t
reject those rows. It stored them as published in year one.
That was harmless by accident. Samvad archived anything older than 90 days, so a year-one article was archived the moment it arrived. It stopped being harmless when I built India News Pulse, which showed who reported a story first. The earliest date wins that, and the earliest date would have been about two thousand years before the story happened.
The fix landed on both sides. The harvester now tries six layouts and falls
back to <lastmod> when the publication date is missing. Pulse stopped trusting
the claim for anything time-sensitive: it ignored dates before 2000 or more than
an hour after we’d first seen the article, and measured how fast a story was
spreading with harvested_at, the one timestamp we record ourselves. One piece
is still open: when neither date parses, the harvester still sends the zero
time.
Some failures look exactly like success. Hand Go’s encoding/xml an HTML
error page or an RSS feed where a sitemap should be, and it decodes into an
empty struct without complaint: zero articles, no error. A source stuck like
that would report nothing on every crawl, with nothing in the logs to say why.
The harvester now reads the root element first and refuses anything that isn’t
<urlset> or <sitemapindex>.
The strangest one parsed perfectly. Another digital-native outlet’s sitemap was valid and up to date, and every URL in it pointed at a domain that no longer resolved. The real articles sat at the same paths on the outlet’s main domain. We ingested 500 rows before noticing none of them had a page behind them, purged them, and dropped the source. The fix is a per-source host rewrite, which is small. I never built it.
One was just too big. The largest sitemap listed about 2,500 URLs against Google’s 1,000. v2 caps a single document at 5,000. That leaves room for every real publisher and still stops one oversized file from holding up every other source’s next crawl.
When the publisher won’t answer
Everything above assumes the publisher answers at all. A few didn’t. The
harvester ran on a GCP VM, and some of the bigger outlets sit behind Akamai’s
bot protection, which scores every request by the reputation of its IP. For
these sites a datacenter address was enough to fail. A GCP address got
403 Access Denied however polite the request was, and a Railway one fared no
better. A fresh VM doesn’t help either, because any new cloud IP lands in
another range with the same reputation.
What those sites don’t block is someone at home on broadband. So I gave the harvester a home connection: an old Android phone on my WiFi, running a proxy app, joined to the VM over Tailscale, which gets through the home router’s NAT without opening any ports. The proxy is a per-source setting, so only the blocked outlets went through the phone and everything else stayed direct. It had to cover both halves of the pipeline. With only the sitemap going through the phone, those outlets’ articles quietly arrived with a title and no description, because the page reads for Open Graph tags were still leaving from the datacenter and getting refused.
The phone was also the least reliable part of the whole system. One morning it was down from 05:57 to 07:13, and every proxied fetch failed with it. I lived with that. Nothing else was affected, and because those sitemaps keep articles for a couple of days, the first crawl after it came back picked up everything it had missed.
One outlet I left alone. Its sitemap sat behind Cloudflare’s “Just a moment” JavaScript challenge, which wants a real browser to run code before it lets anything through. A Go HTTP client can’t do that, and getting past it would have meant running a headless browser for a single source. I dropped it. Another answered 403 even through the phone, so that one went too.
What the harvester is for now
The harvester outlived the product it was built for. What it does now is
narrower and more useful to other people: point it at a list of news sitemaps
and it turns them into a stream of JSON events, one stable ID per article, with
delivery that doesn’t quietly drop anything when a sink is down. Run
docker compose up
and a crawl starts printing to your terminal. You don’t need
an account or any credentials.
It still won’t touch article bodies or feeds. It reads the sitemap and each
page’s <head>, and nothing else.
It ran for three months feeding a product that no longer exists. Out of the box it still crawls every fifteen minutes, for whoever wants to point it at their own list.