Three months of fighting browser automation instability taught us that the real problem was not the tool. It was the architecture. Here is the full story of what broke, what we learned, and how we rebuilt it properly.
There is a specific kind of engineering frustration that comes from spending weeks solving the wrong problem. You patch something, it works for a while, then breaks again in a slightly different way. You patch it again. The cycle repeats. At some point you have to stop and ask whether you are even working on the right thing.
That is more or less what happened when we were building our internal media pipeline at Execross using Playwright. We spent three months fighting browser automation reliability issues before finally accepting that the headless browser approach was fundamentally the wrong choice for what we were trying to do. This post is the full story of what that looked like, what broke, and how we rebuilt it.
How it started
We needed a way to fetch media metadata as part of an internal content archiving tool. The initial implementation used Playwright because it was fast to prototype and did not require understanding any underlying API structure. You navigate to a URL, wait for the page to load, extract what you need from the DOM or intercepted network responses, done.
It worked fine in development. Local machine, fast connection, no real traffic. We shipped it and for the first few weeks everything was fine.
Then it started becoming unreliable. Not completely broken, just unreliable. Roughly one in five requests would fail or return incomplete data. We added retry logic. That helped for a while. Then the failure rate crept back up. We added more sophisticated retry logic with exponential backoff. That helped too, temporarily.
The whack-a-mole phase
What followed was about six weeks of iterative patches, each one addressing a specific failure mode, each one eventually revealing the next problem underneath.
The first thing we tried was varying request timing. Our initial implementation had fairly predictable timing patterns so we added randomized delays between operations. That improved reliability for a couple of weeks.
Then we noticed failures seemed to correlate with session age. We switched to shorter session lifetimes and more aggressive rotation. Again, temporary improvement.
We changed the browser engine entirely. Same result pattern.
Every fix worked for a while and then stopped working. Looking back, this should have been a clear signal that we were treating symptoms rather than the actual problem. Browser automation has a certain inherent fragility because it sits on top of interfaces designed for human interaction, not programmatic access. When those interfaces change, even slightly, automation breaks. And they change constantly, for all sorts of reasons that have nothing to do with you.
The realization
The turning point came when one of our engineers sat down and actually mapped out what the tool was doing at the network level. Playwright was opening a full browser context, loading pages, executing JavaScript, rendering the DOM, and then extracting a handful of fields from a JSON response that the page was fetching in the background.
We were running a full browser rendering engine to extract data from a JSON API call that the page itself was making internally.
That is obviously backwards. The browser was acting as a complicated proxy to an underlying API that we could potentially call directly. The question was what that API looked like and whether we could reach it without the browser involved at all.
Investigating the underlying API
The investigation took about a week. We used mitmproxy to observe the actual network traffic while the browser was doing its thing, then worked backwards from the requests we saw to understand the structure, parameters, and authentication involved.
This kind of analysis of your own tool's network traffic is standard reverse engineering practice. You are not bypassing anything, you are just understanding what your own code is actually doing at the protocol level so you can replicate it more directly and reliably.
What we found was a reasonably clean JSON API underneath. The authentication model was session-based, which meant managing session state, but that is a solved problem with any decent HTTP client library. The actual data extraction was simpler than anything we were doing with DOM parsing in Playwright.
What the rewrite looked like
The rewrite from Playwright to direct HTTP calls took about two days. The resulting code was dramatically simpler. No browser process to manage, no session state tied to a browser context, no DOM parsing, no waiting for JavaScript to execute.
The core of the new pipeline is a thin HTTP client layer with a few specific responsibilities: managing authentication state, handling rate limiting gracefully, parsing the JSON response format, and normalizing output into our internal data model.
The authentication flow initializes a session once, reuses it for as long as it stays valid, and reinitializes automatically when it expires. Session state lives in a simple key-value cache so it survives process restarts.
Rate limiting is handled with a token bucket. We set the refill rate conservatively below what we know the API allows, which means we almost never hit a rate limit response in practice. When we do, we back off with exponential jitter rather than retrying immediately.
import httpx
import time
from typing import Optional
class MediaClient:
def __init__(self, rate_per_second: float = 2.0):
self.session = httpx.Client(timeout=15.0)
self.interval = 1.0 / rate_per_second
self.last_request = 0.0
def _throttle(self) -> None:
now = time.monotonic()
elapsed = now - self.last_request
if elapsed < self.interval:
time.sleep(self.interval - elapsed)
self.last_request = time.monotonic()
def fetch(self, endpoint: str, params: dict) -> Optional[dict]:
self._throttle()
try:
response = self.session.get(endpoint, params=params)
response.raise_for_status()
return response.json()
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
time.sleep(5.0)
return self.fetch(endpoint, params)
return None
except httpx.RequestError:
return NoneThis is a simplified version of the core client. The production implementation has more sophisticated session management and retry logic, but the principle is the same.
Performance difference
The numbers were significant. Our Playwright-based pipeline averaged around 3 to 4 seconds per request including browser startup overhead. The direct HTTP approach averages around 350 to 500 milliseconds depending on network conditions.
Reliability improved even more dramatically. With Playwright we had failure rates that required multiple retries for a meaningful portion of requests. With the direct HTTP approach, failures are almost exclusively network timeouts or rate limit responses, both of which are predictable and handled cleanly.
Memory usage dropped substantially too. A Playwright browser context uses several hundred megabytes of RAM. Our HTTP client uses a fraction of that, which matters when you are running this on a modest VPS.
When headless browsers are actually the right tool
We still use Playwright in a few specific places. Situations where the data we need is only available after client-side JavaScript executes and there is genuinely no underlying API that exposes it. This is less common than you might think, but it exists. Some pages do require JavaScript execution to produce the content we need.
For those cases, Playwright is the right tool. For cases where you are using a browser to access an API that the page is calling internally, switching to direct HTTP is almost always worth the investigation time upfront.
The actual lesson
The engineering lesson here is not about any specific tool. It is about recognizing when you are solving the wrong problem.
If you find yourself adding increasingly elaborate patches to make a fundamentally fragile system reliable, it is worth stepping back and asking whether the architecture itself is the real issue. Sometimes the right move is to spend a week understanding the problem properly rather than spending another few weeks patching symptoms.
In our case, that week of investigation saved us what would have been months of ongoing maintenance. The rewrite was one of the better engineering decisions we made that quarter.