Build a Python Proxy Pool With Retry-on-Block Detection (2026)

Your scraper retries on a 403 and feels safe, but the site is blocking you with a clean 200 that hides a captcha. Your retry logic never fires, your code thinks every page is fine, and you save thousands of junk pages full of challenge html. By the time you notice, the dataset is poisoned and you have to rerun the whole job. Real ban detection reads the body, not the status code.

I run a hardware mobile proxy farm in Singapore. Real phones, real sim cards from Singtel, M1, and StarHub, and I write the Python that drives those proxies every day. So this isn’t theory. This is the same pool code I use to keep jobs alive when a target starts fighting back.

Why the status code lies

The core mistake almost everyone makes is trusting the status code. A clean site returns a 403 or a 429 when it blocks you, and your retry layer catches that and rotates. But the sites that actually hurt you don’t do that. They return a 200 with a soft block, a page that looks normal to your code but is really a captcha wall or a “please wait” screen. So you detect blocks by content signals, and when you find one you rotate the session id to pull a fresh exit ip.

That’s the whole shape of this guide: configure the proxy, rotate by changing one field, detect blocks from the body, then retry with backoff and a hard cap.

Get the proxy config right on line one

In requests you pass a proxies dictionary with two keys, http and https, both pointing at the same proxy url. The url carries your username, password, host, and port in the shape USER:PASS@HOST:PORT. Hand that dictionary to the get call and every request goes out through the proxy.

import requests

proxies = {
    "http": "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=20)
print(r.json())

httpx works almost the same way, but the argument name changed across versions. Older httpx took a proxies dictionary like requests. Newer httpx wants a single proxy argument or a mounts mapping, so check which one you’re on or your config silently does nothing. The trap is httpx will ignore a bad proxies kwarg and send the request from your real ip, so always verify the exit. If you’re new to how these exits work, start with what a mobile proxy actually is.

Rotation is one field, not a new port

With mobile proxies you don’t get a new ip by switching ports. You get it by changing the session id inside the username. Our gateway reads a session token in the user string, and the same token always maps to the same exit ip for as long as you keep using it. That’s a sticky session: you keep one token while you crawl a logged-in flow, then change the token when you want a fresh ip.

So the username looks like your account id, a separator, the word session, then the token. Swap that token for a new random string and the next request leaves from a different sim. Keep it the same and you stay glued to one ip, which matters for any cart or login that hates ip jumps mid-session. One token is one identity, and that’s the whole rotation model in one field.

import uuid

def proxies_for(session_id):
    user = f"ACCOUNT-session-{session_id}"
    url = f"http://{user}:PASS@HOST:PORT"
    return {"http": url, "https": url}

# fresh ip = new token, nothing else changes
fresh = proxies_for(uuid.uuid4().hex)

Getting a fresh ip is just calling that function with a new session id. No new connection pool, no new client config, just a new token.

The block detector reads the body

The block detector is the heart of this whole thing. Write a function called looks_blocked that takes the response and returns true or false. It checks several signals, not one. First the obvious status codes, 403, 429, 503, you keep those. But then the content signals, because those are the ones that save you.

Signal one is body markers. Keep a list of strings that show up on challenge pages: “verify you are human”, the captcha vendor’s name, a known “please wait” phrase, or a hidden form field only the block page uses. If any of those strings is in the response text, it’s a block, even on a 200.

Signal two is body length. A real product page might be eighty kilobytes, but the soft block page is tiny, maybe two kilobytes. So if a page that should be large comes back suspiciously short, that’s a flag.

Signal three is redirects. A lot of blocks bounce you to a /challenge url, a /blocked path, or a login wall you didn’t ask for. In requests the final url sits on the response object after the redirect chain resolves, so read it straight off and compare it to the url you asked for. If they differ and the new path looks like a wall, flag it.

Signal four is the title tag. Challenge pages love titles like “Access Denied”, “Just a moment”, or “Checking your browser”, so a quick check on the title catches a whole family of them.

BLOCK_MARKERS = ["verify you are human", "checking your browser", "just a moment"]
BLOCK_PATHS = ("/challenge", "/blocked", "/captcha")

def looks_blocked(resp, min_len=10_000):
    if resp.status_code in (403, 429, 503):
        return True
    body = resp.text.lower()
    if any(m in body for m in BLOCK_MARKERS):
        return True
    if len(resp.content) < min_len:
        return True
    if any(p in resp.url for p in BLOCK_PATHS):
        return True
    return False

You combine these. Any one of them firing means treat it as blocked.

The strongest signal is the content you came for

The strongest signal is the content you actually came for. If you know the page should contain a price, a product id, or a div that only renders for real visitors, check for that marker directly. Its absence beats any generic string, because it’s tied to the exact page you wanted. So for a target you scrape a lot, write a tiny per-site check for the one element that proves the page is real, and treat its absence as a block.

Retry that rotates and backs off

Now retry with backoff, but smarter than the usual one. The normal pattern just sleeps and retries from the same ip, which is useless if you’re actually banned. My retry does two things on a block: it rotates to a fresh session id so the next attempt leaves from a new exit, and it backs off so you’re not hammering.

The backoff is exponential with jitter. Attempt one waits about one second, attempt two about two, attempt three about four, and I add a small random fraction on top so a swarm of workers doesn’t retry in lockstep.

And you cap it. Pick a max attempts number, say four or five, and once you hit it you stop, mark that url failed, and move on. Without a cap, a permanently blocked url turns into an infinite loop that burns your proxy bandwidth on one dead page.

import random, time, uuid

def fetch(url, max_attempts=5):
    session_id = uuid.uuid4().hex
    resp = None
    for attempt in range(max_attempts):
        proxies = proxies_for(session_id)
        resp = requests.get(url, proxies=proxies, timeout=20)
        if not looks_blocked(resp):
            return resp
        session_id = uuid.uuid4().hex          # rotate the exit
        time.sleep(2 ** attempt + random.random())  # backoff + jitter
    raise RuntimeError(f"blocked after {max_attempts} attempts: {url}")

The loop condition is simple. While attempts is under the cap and the last response still looks blocked, rotate the session, sleep the backoff, and try again. The moment a response passes the detector, break and return it.

Log the exit ip on every attempt

Logging is boring until the day it saves you. On every attempt I log the session id, the exit ip the request actually came from, the status code, and whether the detector flagged it. So when a job goes bad I can read the log and see exactly which exits got blocked and how fast. You can’t tune a pool you can’t see.

How do you even know the exit ip? You ask. Before a big run, and after every rotation, send one cheap request through the proxy to an ip echo endpoint that just returns the ip it sees. Right after you rotate the session id, hit that echo and confirm the ip is different from before. If it didn’t change, your rotation is broken, maybe the token didn’t actually swap, and you fix it before you waste a crawl.

Pace per exit, not globally

Rotation doesn’t excuse hammering. Even with a pool you respect rate limits. I pace per exit ip, not globally, so each sim only sends a request every so often. The idea is the AutoThrottle approach from Scrapy: you watch the response latency and slow down when the server gets slow, because slow responses usually mean it’s straining or starting to throttle you. So you add a delay between requests, add jitter to that delay, and let it grow when latency climbs. A polite pool lives much longer than a greedy one.

Fix the dns leak with one letter

Dns is the leak almost nobody checks. If you use a socks5 proxy the naive way, your machine resolves the hostname locally first, then sends the connection through the tunnel. That means your real resolver, your isp or your office dns, sees every domain you scrape. It can also get you blocked when the dns exit and the http exit don’t match.

The fix is one letter. Use socks5h instead of socks5. The h tells the client to resolve the hostname on the proxy side, so the lookup happens at the exit and never touches your local resolver.

proxies = {
    "http": "socks5h://USER:PASS@HOST:PORT",
    "https": "socks5h://USER:PASS@HOST:PORT",
}

Wrap it in a pool class

All of this should be one object. The pool holds a list of session tokens and hands out a configured client when you ask. Call get_session and it returns a client wired with a proxies dictionary for some token. Call mark_blocked on a token and it rotates that token to a new id and pushes it to the back of the line, so the freshest exits get used first. Your scraper never touches a raw proxy url. It asks the pool for a session, uses it, and reports back when something looks blocked.

Two notes from real jobs. Keep one sticky session per logical task: if you’re crawling one account or cart flow, hold that token for the whole flow, because jumping ip mid-login is itself a block trigger. Across unrelated tasks, spread them over many tokens so no single exit carries all your traffic. And store a small per-token cooldown, so a token that just got blocked isn’t handed out again for a few minutes.

The exits decide everything

The quality of all this depends on the exits underneath it. If your ips are obvious datacenter ranges, the cleanest code still gets soft blocked on the first request. That’s where the farm comes in: real SG mobile ips on actual Singtel, M1, and StarHub sims, sticky sessions you control with the token in the username, and dns through the tunnel with socks5h. If you want to see why this matters at scale, I broke it down across seven web scraping case studies.

If you want to test your pool against exits that actually look like phones, start a free trial of Singapore Mobile Proxy and use code YT30 to get started. Paste your host and port into the proxies_for function above, point your scraper at the pool, and watch the block detector keep your dataset clean.

Get new guides and videos first — join the Telegram channel.