LLM training data proxy for bulk web corpora
Assembling a pretrain or fine-tune corpus means pulling millions of pages from thousands of domains. The moment your crawler runs at that volume from one ASN, the blocks start: 403s, 429s, CAPTCHAs, and the silent soft-bans that quietly poison a dataset with error pages. An LLM training data proxy puts every fetch behind a real mobile IP, so the crawl reaches the end instead of flatlining.
Corpus crawls flatline on IP reputation
Anti-bot systems do not care that your intent is benign — they see request velocity from a datacenter range and throttle it. Mobile carrier exits make each fetch read as a phone user, the traffic these defenses are tuned to allow.
Why training crawls get blocked
Anti-bot systems do not care that your intent is benign. They see request velocity from a datacenter range and throttle it. A web-scale corpus crawl is the most aggressive traffic pattern a site sees, so it trips every rate limit you have. Routing through 4G/LTE carrier IPs makes each request look like an ordinary mobile visitor, which is exactly the traffic these defenses are tuned to allow.
Spread across the pool, sink raw to object storage
For corpus collection you want maximum IP diversity, not session stickiness. Run one worker per IP, rotate each exit on a schedule or on block (at most once per minute), spread the crawl across your pool, and stream raw HTML straight to S3, R2, or GCS. Dedupe by content hash downstream. Our side logs metadata only — no payloads, no inspection of what you collect.
Throughput that matches your worker count
Pretrain crawls run many concurrent workers. Spread them across several dedicated mobile IPs so no single exit becomes the bottleneck. Automatic backoff on 429/503, per-IP cookie jars to keep pagination coherent, and a rotation API (max once per minute per IP, or on your schedule) keep each exit healthy.
Clean data in, clean model out
A blocked request that returns a challenge page is worse than no request — it injects junk into your corpus. Reliable exits mean you collect the actual article, not a "verify you are human" interstitial, so your dedupe and quality filters have real text to work with.
Built for pretrain-scale crawling
Pool-wide diversity
Maximum IP spread across the pool so a million-domain frontier never funnels through one blockable exit.
Worker-matched throughput
Spread concurrent crawl workers across several dedicated IPs; add IPs as the crawl grows.
Raw HTML sink
Stream pages straight to S3 / R2 / GCS and dedupe by content hash downstream — no payload retention here.
Challenge-free fetches
Reliable exits return the real article, not an interstitial, so quality filters get clean text.
Corpus crawl at a glance
LLM training data proxy — questions
What is an LLM training data proxy?+
Why do corpus crawls get blocked?+
How does it scale with workers?+
Does it keep junk out of the corpus?+
Start a corpus crawl
Sizing the crawl
Planning a pretrain-scale pull? Check the pricing for per-IP rates and bulk discounts, then create an account and point your crawler at the rotation endpoint in under 90 seconds.
Pull a sample dataset, free
Run one real mobile IP for an hour with no card. Point your crawler or agent at the source, watch the data come back clean, then move to a plan.