How Do You Run 50,000 Browser Automation Tasks a Day Without It All Breaking?

When I joined Thunder Marketing Corporation, the ask was simple to say and hard to do: automate browser workflows at real scale.

Not hundreds of tasks a day. Tens of thousands. With 99.9% reliability, meaning basically no room for random failures.

Here's how we actually got there.

What we were up against

50,000+ tasks a day. Only about 50 allowed failures in that whole day. Most tasks needed to finish inside 30 seconds. And every task hit a different website with a different structure, any of which could change without warning.

Normal automation scripts fall apart under this kind of load.

The system we built

We split it into four pieces working together: a Redis task queue holding pending work with priority, a pool of Kubernetes workers actually running the browsers, a FastAPI scheduler handing out tasks and retries, and PostgreSQL storing results and logs.

1┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
2│   Task Queue    │────▶│  Worker Pool    │────▶│  Result Store   │
3│   (Redis)       │     │  (Kubernetes)   │     │  (PostgreSQL)   │
4└─────────────────┘     └─────────────────┘     └─────────────────┘
5         │                       │                       │
6         │              ┌───────────────┐               │
7         └─────────────▶│   Scheduler   │◀──────────────┘
8                        │   (FastAPI)   │
9                        └───────────────┘

We started with Selenium, then switched

Selenium worked, technically. But we were constantly writing manual waits, and tests were flaky.

Playwright waits for elements automatically, isolates browser sessions properly, and its trace viewer makes debugging so much easier.

1# Before: manual wait, every single time
2element = WebDriverWait(driver, 10).until(
3    EC.presence_of_element_located((By.ID, "submit"))
4)
5element.click()
6
7# After: Playwright just handles it
8await page.click("#submit")

That switch alone cut our flaky tests by 40%.

Getting to 50K tasks a day

Each worker runs in its own Kubernetes pod, and we scale the pool based on Redis queue depth, not CPU, because this kind of work is mostly waiting on the network, not crunching numbers.

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: browser-worker
5spec:
6  replicas: 20
7  template:
8    spec:
9      containers:
10      - name: worker
11        image: browser-worker:latest
12        resources:
13          requests:
14            memory: "2Gi"
15            cpu: "1000m"

1apiVersion: autoscaling/v2
2kind: HorizontalPodAutoscaler
3metadata:
4  name: browser-worker-hpa
5spec:
6  minReplicas: 10
7  maxReplicas: 50
8  metrics:
9  - type: External
10    external:
11      metric:
12        name: redis_queue_length
13      target:
14        averageValue: 100

Getting to 99.9% reliability

This is the part people underestimate. Not all failures are the same, and treating them the same is how you either give up too early or retry forever.

We classify every failure into one of four types, and only retry the ones worth retrying.

1class FailureType(Enum):
2    TRANSIENT = "transient"      # network hiccup, retry now
3    RATE_LIMITED = "rate_limit"  # back off and wait
4    STRUCTURAL = "structural"    # site actually changed, alert a human
5    PERMANENT = "permanent"      # bad input, don't bother retrying

On top of that, we use circuit breakers so a failing site stops getting hammered:

1from circuitbreaker import circuit
2
3@circuit(failure_threshold=5, recovery_timeout=60)
4async def automate_site_a(task: Task) -> Result:
5    async with async_playwright() as p:
6        browser = await p.chromium.launch()

And selectors that fall back to alternatives when a site changes its HTML:

1class ResilientLocator:
2    def __init__(self, strategies: list[str]):
3        self.strategies = strategies
4
5    async def find(self, page) -> ElementHandle:
6        for strategy in self.strategies:
7            try:
8                element = await page.wait_for_selector(strategy, timeout=5000)
9                if element:
10                    return element
11            except:
12                continue
13        raise ElementNotFound(self.strategies)
14
15submit_button = ResilientLocator([
16    "#submit-btn",
17    "button[type='submit']",
18    "text=Submit",
19])

When even that fails, we use AI as the last resort

For sites with genuinely obfuscated or randomly generated class names, standard selectors just don't work. We fall back to asking GPT-4 Vision to look at a screenshot and point at the right element.

1async def find_element_with_ai(page, description: str):
2    screenshot = await page.screenshot()
3
4    response = await openai.chat.completions.create(
5        model="gpt-4-vision-preview",
6        messages=[{
7            "role": "user",
8            "content": [
9                {"type": "text", "text": f"Find the {description} element"},
10                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{screenshot}"}}
11            ]
12        }]
13    )
14
15    coordinates = parse_coordinates(response)
16    await page.click(position=coordinates)

It's slower and costs more, so we only reach for it when everything else fails.

You have to be able to see what's happening

At this scale, you cannot maintain 99.9% reliability blind. We track task counts, latency, and queue depth with Prometheus, and alert automatically when failure rate crosses 1% over 5 minutes.

Where we landed after 9 months

50K+ tasks daily, consistently. 99.9% success rate, usually 30 to 40 failures in a whole day. P95 latency under 25 seconds. 60% cheaper than doing it manually. And reliability improved 85% compared to our first version.

The honest takeaway: pick tools that handle waiting for you, design for things to fail gracefully instead of pretending they won't, and scale on the bottleneck that's actually real, which for browser work is almost always I/O, not CPU.

FAQ

Should I use Selenium or Playwright for browser automation at scale?

Playwright. Auto-wait and proper isolation alone cut our flaky tests by 40%.

How do you keep browser automation reliable at scale?

Classify failures and only retry the ones worth retrying. Alert a human when a site's structure actually changed instead of retrying forever.

Do you scale browser workers based on CPU?

No, scale on queue depth. This kind of work is I/O bound, so CPU usage doesn't tell you much.

Building automation systems? Let's connect on LinkedIn or check out my work on GitHub.