Scraping at Scale Without Getting Blocked: The Real Architecture 2026
Scraping at Scale Without Getting Blocked: The Real Architecture 2026
A scraper that works beautifully on a hundred pages and falls apart on a hundred thousand is not a code problem, it is an architecture problem. The script is fine. What breaks is everything around it: the pacing, the rotation, the way it handles being told no.
I run a hardware mobile proxy farm in Singapore, real phones on real Singtel, M1, and StarHub sims, and a lot of scrapers exit through it. So I get to watch which designs scale cleanly and which get walled within an hour. The difference is almost never the parser.
Here is the architecture in one line, and then I will take it apart. A queue of work, a rate limit applied per exit ip, rotation across mobile exits, and retries with backoff when you get blocked. The proxy is one layer in that stack, not the whole answer. People buy good proxies, point a reckless scraper at a target, get blocked, and blame the proxies. The proxies were fine, the architecture was not.
Start with a queue, not a loop
Scale begins at the queue. You do not want a script that loops through a list firing requests as fast as the cpu allows. You want a queue of jobs and a set of workers pulling from it at a controlled rate. That lets you pause, resume, retry, and reprioritise without losing your place, and it turns throughput into a dial you set rather than whatever the machine happens to do. Once work lives in a queue, scaling is tuning, not rewriting.
[ job queue ] --> [ worker pool ] --> [ rate limiter per exit ip ]
|
v
[ rotate across mobile exits ]
|
success <----------+----------> block detected
| |
validate body backoff + requeue
The one number that matters: requests per ip per minute
The single most important number in the whole system is requests per ip per minute. Every exit ip has a budget of how much traffic it can send to a target before the target notices and reacts, and your job is to stay under that budget on every ip. Blast one ip with a thousand requests a minute and it gets blocked no matter how good it is. So you set a per ip rate limit, a polite ceiling, and you never cross it on any single exit.
This is the mental shift that makes scraping scale. You do not go fast by making one connection aggressive, you go fast by spreading a gentle load across many exits. Ten ips each making a calm number of requests beats one ip making a frantic number, every time, because the calm ones never trip the defence. Scale is parallel politeness, not concentrated aggression.
Match your rotation strategy to the job
How you assign ips to requests matters as much as how many you have. There are three common patterns and they suit different jobs.
- Per request rotation gives every request a fresh exit. That is great for stateless scraping where each page is independent and you want maximum spread. The downside is you build no session, so it is wrong for anything that needs continuity.
- Per session rotation holds one exit for a sequence of related requests, then switches. You use this when a site expects some continuity, like paging through results that track a session. You look like one visitor for a while, then become a different one. It balances spread against the need to not look like you teleport mid session.
- Per target rotation keeps pools separated by destination. If you are scraping several sites, you do not want one ip’s behaviour against site a to affect its standing on site b, and you want to tune rate limits per target since each tolerates a different pace. A slow strict target gets gentler pacing than a permissive one.
On my farm you can pull a fresh exit per request or hold one with a sticky session, which is exactly what these patterns need underneath them.
Detect blocks, including the soft ones
At scale you will get blocked, and how you respond decides whether you recover or spiral. The first job is to actually detect a block, and it is not always obvious. A hard block is a clear status code, a 403 or a 429 telling you to slow down or go away. But the nastier ones are soft blocks. The request returns a 200 and looks fine, but the content is a captcha page, or an empty result, or a decoy. If your scraper treats those as success, you are storing garbage at scale.
So you validate responses, not just status codes. You check that the page actually contains the data you expected, and you treat a missing element as a block even when the status says ok.
def is_block(resp):
if resp.status_code in (403, 429):
return True
body = resp.text.lower()
if "captcha" in body or "are you a robot" in body:
return True
# soft block: 200 OK but the data we came for is missing
if not extract_target_field(resp):
return True
return False
Once you can tell a real block from a real page, you can respond correctly. Blind scrapers that only look at status codes fill their database with captcha pages and nobody notices until the data is useless.
Back off, do not brute force
When you do hit a block, the response is backoff, not brute force. You do not immediately retry the same ip harder, because that is exactly what makes it worse. You back the ip off, rotate to a fresh exit for the work, and retry the failed job later with increasing delays. Exponential backoff means each repeated failure waits longer before trying again, which lets a stressed target and a stressed ip both cool down. A scraper that retries instantly just digs its own hole deeper.
import time
def fetch_with_backoff(job, max_tries=5):
for attempt in range(max_tries):
exit_ip = rotate_exit()
resp = fetch(job.url, proxy=exit_ip)
if not is_block(resp):
return resp
cool_down(exit_ip) # rest this ip
time.sleep(2 ** attempt) # 1s, 2s, 4s, 8s, 16s
requeue(job) # set aside, keep the run flowing
A 429 specifically is the target telling you the rate is too high. The correct response is to lower your per ip rate, not to switch ip and keep the same reckless pace, because you will just burn the next ip the same way. Treat a 429 as feedback about your pacing globally, not just a problem with the one ip that hit it. The target is teaching you its budget. Listen.
Why mobile exits survive where datacenter pools get walled
This is the part the proxy actually decides. A datacenter ip is easy to identify by its asn, and many targets distrust hosting ranges by default, so a big datacenter pool can get walled wholesale, entire ranges blocked at once. A mobile exit resolves to a carrier asn and shares its address with crowds of real humans, so the target cannot block it without collateral damage to real users. That reluctance is exactly the survivability you are buying. If you want the longer version of why this holds across real jobs, I went through it in web scraping with mobile proxies: 7 case studies.
It also means the same polite request budget goes further on mobile. On a distrusted datacenter ip you get a tiny amount of rope before you are blocked. On a trusted mobile ip you get more room within the same careful pacing, because the baseline trust is higher. You still have to be polite, the ip type does not excuse aggression, but politeness on a trusted network buys you far more than on a suspect one. The difference between a residential and a mobile proxy is part of this same trust story.
Do not leak dns
Scrapers leak dns constantly. If your domain lookups go out through your own machine’s resolver while your requests exit through a Singapore mobile ip, you have told the destination a contradictory story about where you really are.
The lookups should route through the same tunnel as the traffic, so the resolver and the exit agree. A consistent story holds up at scale, a leaky one quietly flags you no matter how good your pacing is. In practice that means a socks5h scheme so the proxy resolves the hostname, not your machine.
Concurrency is width, not depth
People confuse concurrency with speed. Concurrency is how many requests are in flight at once, and it is tempting to crank it up to go faster. But concurrency without per ip limits just means you hit one target harder in parallel, which gets you blocked in parallel. The right way to think about it is concurrency across exits. If you have twenty exits each allowed a gentle rate, your total concurrency is the sum of all those gentle streams, and that is genuinely fast while staying polite on every ip. Width, not depth.
Match the request layer to the network layer
A perfect proxy with an obvious bot signature still gets caught. A real browser sends a coherent set of headers, a believable user agent, the right accept language, a referer that makes sense. A default script library sends a giveaway fingerprint. So at scale you make your requests look like real clients, and vary that across exits so a thousand identical fingerprints do not all arrive from your pool at once. The network and request layers have to tell the same story.
Cache, isolate failures, and monitor
Three architecture decisions people skip carry the run at scale. The first is caching. If you are pulling a hundred thousand pages, some have not changed since last time, and re scraping them wastes ip budget on nothing. Track what you have, respect last modified where it exists, and spend requests only on what is new. The cheapest request is the one you skipped.
The second is failure isolation. One target going down or one ip getting blocked should not stall the entire run. Workers should fail a job, set it aside, and keep pulling other work. At a hundred thousand requests something is always failing somewhere, so design for partial failure as the normal state and the run flows around the rough spots.
The third is monitoring, which turns all of this from hope into control. You want to see, live, your success rate, your block rate per target, your throughput, and how each exit is doing. When the block rate on a target climbs, that is your warning to slow down before you get walled, not after. A scraper that behaves like a swarm of polite visitors lasts. One that behaves like an attack gets treated like one.
The failure pattern I see most
Someone builds a fast scraper, points it at a single good proxy, runs it flat out, gets blocked in twenty minutes, and concludes proxies do not work. The real problem was no queue, no per ip rate limit, no backoff, and one ip doing all the work. Add those four things and the same target that walled them in twenty minutes becomes scrapable for as long as they like. The architecture, not the ip, was the wall.
The real shape of scraping at scale
So that is the real shape of it. A queue you control, a strict per ip rate limit, rotation matched to the job, response validation that catches soft blocks, backoff that cools things down instead of escalating, and trusted mobile exits with consistent dns underneath it all. The proxy is one essential layer, not the whole machine.
If you want that layer done right, that is what I run: real Singapore mobile ips on actual Singtel, M1, and StarHub sims, clean rotation, sticky sessions when you need continuity, and dns that routes through the tunnel so you stop leaking your real resolver. Start a free trial of Singapore Mobile Proxy and use code YT30 to get going, then point your queue at the exits.
Get new guides and videos first — join the Telegram channel.