Four ways to reach a page that looks dead
A blocked request is not a dead page. Work through the ladder before you drop a source.
6 min read
Curious
0 XPA lot of legitimate, perfectly live sites refuse plain automated requests. Major consultancies, government statistics sites, academic publishers, review platforms, Reddit, and most large vendors.
If you treat a refusal as a dead page, you will drop your best sources and keep the ones that were easy to fetch. That is a bad trade, and it shows in the article.
The ladder
| # | Method | Gets past | Watch for |
|---|---|---|---|
| 1 | Plain fetch | Nothing much | A 403 here means almost nothing |
| 2 | Reader API | Most bot blocks | A tiny response that is really a tracker pixel |
| 3 | WebFetch | Some of what stops the first two | Different client, different result |
| 4 | Screenshot it, then read the image | Almost everything | This is the one that works |
Step four surprises people. Screenshotting a page and then actually looking at the image works when all three fetch methods fail, because the page is rendered by a real browser.
The block that looked like an empty page
Two press releases from a major analyst firm were needed as sources. Plain fetch: 403. WebFetch: 403. The reader API returned something worse than an error, a one-pixel tracking image with the text 'A 1x1 image, likely be a tracker probe'. That looks like a page that is empty or needs JavaScript, not a page that is blocking you. A screenshot rendered both releases in full, including the figures, the sample size and the analyst quote.
Cost: Two strong sources were nearly dropped as unreachable.
The rule that came out of it. Run all four rungs before calling a source unreachable. And treat a suspiciously tiny response as a block wearing a disguise, not as an empty page.
The one failure that is real
A DNS failure or a total non-response is different from a block, and it does disqualify a source. Never cite a figure from a page you could not reach by any method. "It was probably fine" is how invented statistics enter an article.
Do this now
- 1
Add the ladder to your CLAUDE.md as four lines.
Before calling a source unreachable, try in order: curl -> reader API -> WebFetch -> screenshot the page and READ the image. A 403 is not death. A DNS failure is. Never cite a figure from a page you could not reach.
- 2
Test it on a site that blocks bots. Pick any large consultancy report page.
Watching rung one fail and rung four succeed once is worth more than reading about it.
Finished reading?
Mark it done to bank the XP and keep your streak alive. Any challenges on this page score separately, so you can come back for a perfect run later.