Using Singapore mobile proxies with Scrapy in 2026

At some point in a serious Scrapy project, you stop tuning the spider and start thinking about your IP strategy. The spider is fine. The concurrency settings are dialed in. The item pipelines are clean. But somewhere between the request going out and the response coming back, a wall appears: a 403, a CAPTCHA farm, a redirect to a “suspicious activity” page, or just silent data corruption where the site serves a degraded view without ever saying so. For engineers targeting Southeast Asian platforms, SG-specific e-commerce, or any service that fingerprints by carrier and geography, mobile proxies on real Singapore carrier IPs are the answer that actually holds up in production.

why Scrapy hits walls without residential mobile IPs

The detection surface for a Scrapy crawler running through datacenter IPs is enormous. ASN reputation is the first check every serious anti-bot system runs. Datacenters like AWS ap-southeast-1, DigitalOcean SGP1, or commercial proxy network ranges are well-indexed in threat intelligence feeds. The moment your IP resolves to an ASN owned by a hosting provider, you are already in the “treat with suspicion” bucket before a single HTTP header gets examined.

From there it compounds. Scrapy’s default concurrency model sends many requests from the same IP in a short window. Even with AUTOTHROTTLE_ENABLED and conservative DOWNLOAD_DELAY settings, the request cadence from a single datacenter IP looks nothing like organic mobile browsing traffic. Sites that have invested in behavioral analysis (Cloudflare, PerimeterX, Akamai Bot Manager) score these sessions low immediately. You can mask the user-agent, randomize headers, and add realistic accept-language strings, but the IP-level signal overwhelms the header-level mimicry.

There is also a subtler problem specific to Scrapy workflows: the tool makes it very easy to run parallel spiders or use CrawlSpider to fan out across hundreds of URLs quickly. That velocity, even when throttled, triggers rate-limit logic at the CDN layer keyed to IP prefix rather than individual IPs. You rotate IPs and land in the same /24, and the rate limit still applies. Residential mobile IPs, assigned dynamically by a carrier’s CGNAT pool, do not share that problem. Each IP is genuinely distinct, genuinely mobile, and carries no prior reputation baggage. To understand the fundamental difference in how these IP types behave, HTTP vs SOCKS5 mobile proxies covers the protocol layer in detail, which matters when you are choosing which endpoint type to configure in Scrapy’s middleware stack.

setting up SMP credentials in Scrapy

Singapore Mobile Proxy provides credentials in the format host:port:username:password. You get a host (the proxy gateway), a port number, and per-account or per-session credentials. The cleanest way to wire this into Scrapy is through a custom DownloaderMiddleware rather than Scrapy’s built-in HttpProxyMiddleware, because you want full control over credential rotation and session handling without fighting the built-in middleware’s assumptions.

Here is a working baseline middleware you can drop into your project’s middlewares.py:

import base64
from scrapy import signals
from scrapy.exceptions import NotConfigured

class SMPProxyMiddleware:
    """
    Injects Singapore Mobile Proxy credentials into every request.
    Configure in settings.py:
        SMP_PROXY_HOST = "your.smp.gateway"
        SMP_PROXY_PORT = 10000
        SMP_PROXY_USER = "your_username"
        SMP_PROXY_PASS = "your_password"
        SMP_PROXY_SCHEME = "http"  # or "socks5"
    """

    def __init__(self, host, port, user, password, scheme):
        self.proxy_url = f"{scheme}://{host}:{port}"
        credentials = f"{user}:{password}"
        encoded = base64.b64encode(credentials.encode("utf-8")).decode("utf-8")
        self.proxy_auth = f"Basic {encoded}"

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        host = settings.get("SMP_PROXY_HOST")
        if not host:
            raise NotConfigured("SMP_PROXY_HOST not set")
        return cls(
            host=host,
            port=settings.getint("SMP_PROXY_PORT", 10000),
            user=settings.get("SMP_PROXY_USER", ""),
            password=settings.get("SMP_PROXY_PASS", ""),
            scheme=settings.get("SMP_PROXY_SCHEME", "http"),
        )

    def process_request(self, request, spider):
        request.meta["proxy"] = self.proxy_url
        request.headers["Proxy-Authorization"] = self.proxy_auth

In settings.py, disable the built-in proxy middleware and enable yours:

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
    "myproject.middlewares.SMPProxyMiddleware": 350,
}

SMP_PROXY_HOST = "your.smp.gateway"
SMP_PROXY_PORT = 10000
SMP_PROXY_USER = "your_username"
SMP_PROXY_PASS = "your_password"
SMP_PROXY_SCHEME = "http"

Do not hardcode credentials in settings.py in a real project. Pull them from environment variables using os.environ.get("SMP_PROXY_USER") or from a secrets manager. The structure above makes that substitution straightforward.

One thing to check: if you are using SOCKS5 endpoints, Scrapy requires the PySocks package and your proxy URL must use the socks5:// scheme. HTTP endpoints work out of the box. For most Scrapy use cases targeting HTTPS targets, HTTP CONNECT proxying (the default HTTP proxy mode) handles TLS traffic cleanly, so SOCKS5 is only necessary if a specific target or middleware chain requires it.

rotating IPs per request or per session

The decision between rotating and sticky sessions is not about which is “better” in the abstract. It comes down to what your target site’s session model requires.

For pure data extraction where each request is stateless (product listing pages, public search results, price feeds), per-request rotation is the right call. Every request gets a fresh IP from SMP’s carrier pool, maximizing coverage and minimizing any individual IP’s exposure to rate-limit counters. This is the mode that maps most naturally to Scrapy’s concurrent request model.

For workflows involving login flows, cart interactions, multi-step forms, or any server-side session that ties a cookie to a source IP, you need sticky sessions. Rotate IPs mid-session and the target site sees a session cookie arriving from a different IP than it was issued to, which triggers immediate invalidation or a re-auth challenge on most platforms.

SMP provides both modes through its endpoint configuration. The rotation behavior is controlled at the endpoint level, not by anything you change in Scrapy. What you control in Scrapy is whether you persist a single proxy configuration across a whole session or swap it out per-request.

Here is how to implement per-request IP rotation cleanly in the middleware, pulling a fresh credential set or endpoint for each outgoing request:

import random
import base64

class SMPRotatingProxyMiddleware:
    """
    Rotates across a list of SMP proxy endpoints per request.
    Each entry in SMP_PROXY_LIST is a dict with host, port, user, pass, scheme.
    """

    def __init__(self, proxy_list):
        self.proxy_list = proxy_list

    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.getlist("SMP_PROXY_LIST")
        if not proxy_list:
            raise NotConfigured("SMP_PROXY_LIST not set")
        parsed = []
        for entry in proxy_list:
            # entry format: "scheme://user:pass@host:port"
            parsed.append(entry)
        return cls(parsed)

    def process_request(self, request, spider):
        proxy_entry = random.choice(self.proxy_list)
        # proxy_entry is already in scheme://user:pass@host:port format
        # Scrapy handles auth extraction from the URL automatically
        # when using this format with HttpProxyMiddleware.
        # If rolling your own, parse and set meta["proxy"] and the auth header.
        request.meta["proxy"] = proxy_entry

For sticky session workflows, the cleaner pattern is to assign a single proxy endpoint at the spider level (or at session-start in a start_requests override) and propagate it through request.meta["proxy"] by passing it explicitly in cb_kwargs or attaching it to a session object your middleware reads. Avoid letting sticky-session requests flow through a rotation middleware.

three real workflows where this combo wins

scraping SEA e-commerce search rankings

Price intelligence and organic search rank tracking on platforms like Lazada SG, Shopee SG, or Qoo10 depends entirely on seeing the same result a real Singapore mobile user sees. These platforms serve location-personalized results based on a combination of IP geolocation and carrier detection. A datacenter IP in the same AWS region often resolves to a business ASN and gets a different content tier, sometimes subtly (slightly different pricing, different sponsored slot injection) and sometimes dramatically (full redirect to a regional homepage). Running Scrapy through SMP’s SingTel or StarHub IPs means your crawled data reflects actual mobile consumer-facing output, which is the data point that matters for the business decision. This use case is covered in more depth in mobile proxies for SEO research.

monitoring geo-fenced app content and API responses

Several Singapore government-adjacent platforms, fintech apps, and media services implement geo-fencing at the carrier level, not just IP geolocation. They check whether the connecting IP belongs to a licensed Singapore operator. Scrapy jobs scraping API responses, app landing pages, or compliance-relevant content surfaces hit hard blocks when connecting from foreign or datacenter IPs, even if those IPs are geographically tagged to Singapore. SMP’s real carrier IP pool (SingTel, StarHub, M1, Vivifi) passes these carrier-level checks because the IPs genuinely belong to those operators. No IP spoofing or geolocation header manipulation is involved, which also matters from a legal and ethical mobile proxy use standpoint.

Account-based scraping (collecting data that requires a logged-in session, specifically your own account’s data across multiple accounts, not scraping private data you are not authorized to access) is a workflow where Scrapy’s FormRequest and cookie middleware combined with sticky SMP sessions produces much better outcomes than datacenter IPs. Assign one sticky proxy endpoint per account session, keep that assignment for the life of the session, and let Scrapy’s cookie jar handle the session cookies normally. The target site sees consistent IP-to-session pairing throughout, which is exactly what a legitimate mobile user looks like. This avoids the re-auth loops and temporary locks that plague the same workflow when run through rotating datacenter proxies.

common pitfalls

User-agent and IP type mismatch. If you are routing through a mobile carrier IP but sending a desktop Chrome or Python-requests user-agent, some bot detection systems flag the mismatch. Set your user-agent to a realistic mobile browser string when using mobile proxy endpoints.
Rotating IPs through session-sensitive flows. This is the single most common integration mistake. If you enable per-request rotation and your spider has any login, cart, or multi-step flow, expect session invalidation. Explicitly scope rotation to stateless requests only.
Ignoring 407 responses. HTTP proxy authentication failures return 407, not 403. Scrapy’s default retry middleware does not treat 407 as retryable. If your credentials are wrong or expire, requests fail silently unless you handle 407 explicitly in your middleware or in a custom process_response.
AUTOTHROTTLE masking real proxy errors. With AUTOTHROTTLE_ENABLED, Scrapy slows down when it sees errors. If your proxy is misconfigured, the spider will appear to run but will crawl at near-zero throughput while autothrottle backs off indefinitely. Check your proxy response codes in the stats output (downloader/response_status_count/200, etc.) early in a run.
Not accounting for proxy latency in timeout settings. Mobile proxy gateways add round-trip latency compared to direct connections or datacenter proxies. If your DOWNLOAD_TIMEOUT is set aggressively low (under 15 seconds), you will see spurious timeouts that look like target site failures but are actually proxy latency. Raise the timeout and confirm actual 200 response times before tuning throttle settings.
Running identical Scrapy jobs from a single IP concurrently. Even with mobile IPs, hammering a single endpoint with high concurrency defeats the purpose. Use Scrapy’s CONCURRENT_REQUESTS_PER_IP setting and spread load across multiple proxy endpoints if you need high throughput.

when Singapore IPs specifically matter

The SEA proxy market is full of “residential” offerings that are US or EU residential pools re-labeled as “including APAC.” For a Scrapy workflow targeting Singapore-specific platforms, that distinction is not minor. Singapore-based services implement geolocation checks at multiple layers: IP geolocation databases, carrier ASN lookup, and in some cases latency fingerprinting that distinguishes a locally-originating request from one routed through an offshore proxy claiming a Singapore exit node. Real carrier IPs from SingTel, StarHub, M1, and Vivifi sit in the right ASN, have correct PTR records, and produce the right latency profile for locally-originating requests.

There is also the market data accuracy angle. If your Scrapy jobs are collecting pricing, inventory, or ranking data for Singapore market analysis, you need to see exactly what a Singapore mobile consumer sees. Platforms increasingly segment their served content by IP carrier, not just country. A foreign residential IP that happens to geolocate to Singapore will see a different content tier on Shopee SG or Lazada SG than a genuine SingTel mobile IP. The difference in data fidelity between those two scenarios is not a rounding error. It is the difference between collecting accurate market intelligence and collecting a datacenter-flavored approximation of it.

getting started

If you are ready to wire this up, the Singapore Mobile Proxy plans) page lists current plan tiers with HTTP and SOCKS5 endpoint options and both rotating and sticky session configurations. For background on how mobile proxies differ from the residential pool products you may have used before, what is a mobile proxy is a useful reference. The integration code above is production-ready as a starting point; the main thing left is tuning CONCURRENT_REQUESTS, DOWNLOAD_DELAY, and your retry logic to match the throughput profile of your specific target.