Web scraping at scale with mobile proxies (Playwright + rotation)

Your scraper works fine until it doesn’t. It pulls page one, ten, a hundred, then somewhere past a few thousand requests the blocks start. Captchas, empty bodies, connections that just hang. The code didn’t change. The volume did. This is the part of scraping nobody warns you about, and a web scraping mobile proxy is one of the few things that actually fixes it at scale.

Why datacenter proxies fail at scale

Datacenter proxies are cheap, fast, and easy to spot. They live in known IP ranges owned by hosting companies, and every serious anti-bot system has those ranges mapped. One request from a datacenter IP is fine. A thousand requests an hour, all from the same handful of ranges, with no human-looking traffic around them, is a pattern. Sites flag the ASN, not just the IP, so rotating to the next datacenter IP in the same block buys you nothing.

The deeper problem is that there’s no cover. A real site gets traffic from thousands of normal phones and home connections. Your datacenter IPs don’t blend into that. They stick out as exactly what they are: a server pretending to be a person.

When you actually need mobile

A mobile IP looks like a real phone on a real carrier. In our case that’s a Singapore SIM on Singtel, M1, or StarHub, sitting behind carrier grade NAT. That last part matters: carrier NAT means a single public IP is shared by lots of real subscribers at once. Blocking that IP means blocking real customers, so sites are far more cautious about it. Your scraper hides inside genuine human traffic instead of standing apart from it.

You need mobile when the target is aggressive, when you’re hitting it at volume, or when the data sits behind a login. For everything else, you might not. More on that below.

The Playwright proxy config

Playwright takes proxy settings right in the launch call. Point the browser at the proxy host and port, then pass the username and password:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "socks5://gateway.example.net:1080",
        "username": "your-user",
        "password": "your-pass",
    })
    page = browser.new_page()
    page.goto("https://ipinfo.io/json")
    print(page.content())
    browser.close()

That’s the whole hookup. The host stays constant; the IP behind it rotates on the carrier’s schedule. You talk to the gateway, not a raw phone IP, so a rotation doesn’t break your session.

socks5h and DNS

Use socks5h, not plain socks5. The extra h tells the client to resolve DNS through the proxy tunnel instead of on your own machine. With plain socks5, your local resolver looks up the hostname before the request leaves, which leaks your real location and undercuts the whole point of the proxy. Browser automation can route DNS oddly, so check it: hit an IP-check endpoint through the proxy and confirm both the IP and the DNS line up with Singapore before you trust the setup.

Sticky vs rotating per task

Pick the session type to match the task, not by habit. A sticky session holds one IP for a set window, which is what you want for anything multi-step: a login, a checkout, a flow where the site expects the same person across several requests. If the IP changes halfway through, the site sees a session that teleported to a new address, and that’s a red flag.

Rotating is for high-volume fetches where each request stands alone. Scraping a list of product pages, pulling search results, grabbing one URL after another with no shared session. There, a fresh IP per request or per small batch spreads the load and keeps any single IP from looking busy.

Concurrency caps per IP

This is where most scaling attempts die. A mobile proxy is one phone with one radio. Fire forty parallel threads through it and you’re asking a handset to behave like a server. It starts refusing and dropping connections, which looks exactly like an unstable proxy when it’s really you overloading one device. Keep concurrency per IP to a handful, not dozens. If you need more throughput, you need more devices, not more threads through one.

Retries and backoff

Mobile connections blink. A SIM hands off between towers, the signal dips, and the data connection drops for a second or two. That’s a real radio on a real network, and it’s unavoidable. So your client has to expect it. Wrap each request in a single automatic retry with a short wait, and use exponential backoff when you hit repeated failures: wait a little, then a little more, instead of hammering straight back. Carrier NAT also drops idle entries, so on long-lived connections send a small keepalive every thirty to sixty seconds to keep the tunnel warm.

Detecting soft blocks

The hardest blocks to debug are the quiet ones. A site that’s onto you often won’t return a clean error. It serves an empty body, a stripped-down page, a captcha, or just lets the connection hang until it times out. From your side that can look like a flaky proxy, so you switch providers and see the same thing, because the proxy was never the problem.

The test is simple. Hit a neutral endpoint like ipinfo.io through the same proxy in a tight loop. If that’s rock solid but your target keeps dropping or returning junk, the target is blocking you, not the proxy. Then the fix is slower pacing and fresh IPs, not a new provider.

Realistic request pacing

Speed is what gets you caught. Real users don’t fire requests every fifty milliseconds with machine precision. Add delays between requests, vary them a little, and stay under whatever rate the site tolerates. It feels slow, but a steady scrape that finishes beats a fast one that gets your IPs burned in the first hour. Pacing plus rotation plus a believable IP is the combination that holds up over thousands of requests.

You might not need mobile

Honest note: mobile isn’t always the answer. If you’re scraping public data with no login, at light volume, off a site that doesn’t fight back, datacenter or residential proxies are cheaper and plenty. Mobile earns its cost on hard targets, high volume, and account-bound data. Match the tool to the job. There’s a deeper breakdown in web scraping with mobile proxies: 7 case studies, and if the basics are new, what is a mobile proxy covers the ground first.

Scaling across devices

The right way to scale isn’t more threads through one phone, it’s more phones. Spread your concurrency across many devices, each on its own clean line, each carrying a small share of the load. That keeps every individual IP looking like a normal subscriber while your total throughput climbs. It also means one device blinking or rotating never stalls the whole run, because the work is already distributed.

The latency tradeoff

Be realistic about speed. Mobile latency runs roughly eighty to three hundred and fifty milliseconds, well above a datacenter connection. That’s the cost of looking like a real phone, and it caps per-IP throughput. You make it back with parallelism across devices and with the simple fact that a slower scrape that doesn’t get blocked finishes, while a fast one that gets banned doesn’t. Plan for throughput across the fleet, not raw speed per request.

Try it on real SG mobile proxies

If your scraper has hit the wall, this is exactly what mobile proxies are for. Singapore Mobile Proxy runs a real hardware farm: genuine SG SIMs on Singtel, M1, and StarHub, sticky or rotating, one clean line per task. There’s a free trial so you can point Playwright at a real SG mobile IP and see whether your blocked target behaves. Use code YT30 at signup. Test it on the site that’s been fighting you and find out before you commit.

Get new guides and videos first — join the Telegram channel.