Giving Coding Agents a Logged-In Browser

A coding agent that changes a web page needs to see the page. Unit tests and a successful build say nothing about whether a button is visible, whether a dialog closes, or whether the deployed site is running the new code. So our agents get a real browser, driven through Playwright, and they use it after every UI change goes live.

Getting that to work reliably for many agents at once took three months and 38 changes to one module. Most of the work went into the cases where an agent drew the wrong conclusion from what the browser showed.

How it is set up

• One long-lived browser per profile. The first browser command starts a background Playwright process. Later commands reuse it, so the page, tabs and login carry over between separate commands, the way they would for a person. It shuts down after 15 idle minutes.
• Named profiles. Each profile has its own browser process, cookies, storage folder and port. Agents verifying different things use different profiles and do not share login state.
• Login from the vault. A login command signs in headlessly through our identity provider with credentials fetched from the password vault, saves the browser session to a file, and passes that file to the next command. The agent never sees the password.
• Scripts as well as clicks. For checks with several steps, agents send a Playwright script to the running browser instead of issuing one click at a time. The script is sent once and never replayed automatically, because replaying a script that submits a form can submit it twice.
• Evidence bundles. One command captures a screenshot, the page structure, console errors and network requests together, so a report can show what the agent saw.

Mistakes agents made, and what changed

Each of these happened in practice. In most of them the browser worked correctly; the agent misread what it showed.

“The site is down”

Agents can attach to a browser running on the operator’s own machine, through a tunnel, which is useful for sites that challenge automated browsers. When that tunnel failed to resolve, agents reported that the site itself was unreachable. The standing instructions now say it plainly: a tunnel or DNS error is not evidence that the app is down. Check the app another way first.

A related case: a browser window on the operator’s machine was hidden behind other windows, so the browser stopped rendering it. A bot challenge on the page showed only its label, without the checkbox. To the agent, it looked like the site had blocked access, and several tasks stalled on it. The fix tells the browser to treat the page as visible and focused, so it keeps rendering.

“The browser isn’t available”

Agent machines ship Chromium rather than Google Chrome. An older version of our tool looked for Chrome, which then crashed on launch, and agents concluded they could not inspect the site at all. The tool now finds the installed Chromium and adds the flags it needs to run as root inside a container. The instructions now say a missing Chrome error is a setup problem to fix, not a reason to skip verification.

“The button doesn’t exist”

An agent asked about a control on a page searched the source code on its branch, did not find it, and concluded that the control belonged to some other product. The page it was asked about was running a different build from the agent’s checkout. The rule now: look at the deployed page and compare its build with your checkout before claiming something does not exist.

Comparing builds has its own trap. During one verification, the site’s build marker file reported one revision while the JavaScript actually loaded in the browser came from another, even after a reload. The agent caught it by checking the revision embedded in the loaded bundle as well as the marker file. The loaded code is what the user runs.

Logged out without knowing it

A profile name does not log you in; only the login command does. Early on, agents opened a named profile, saw a page, and reported on it, without noticing it was the signed-out version of the page.

A subtler version came later. The background browser restarts after a crash or the idle timeout. The saved login was only handed to the command that ran right after signing in, so a restart brought the profile back signed out, and every later command ran against a signed-out session. Now each profile remembers which login it was started with and restarts with it. If it is handed a different login, it restarts instead of continuing with the old one.

The wrong tool name, twelve times

The Playwright process we run offers its script tool under a name that differs from the one in most documentation. Agents kept calling the documented name and getting “tool does not exist”: twelve times in one review of agent logs. The fix was a single stable command that asks the browser process which script tool it actually offers and uses that one.

Testing that fix in a real browser before merging found a second bug: requests to the local browser process were going through the machine’s outbound proxy, which rejected them. A mocked test would not have found it.

Screenshots are not enough

For our app pages, agents preview a new design with realistic data and must check desktop and mobile screenshots, the empty, loading and error states, overflow, keyboard focus and the main action. That list exists because a page can pass every schema check and still look broken, and because a page that looks right in one state often breaks in another. Even then, a preview does not prove the data is saved or that the server enforces its rules. That still needs a check against the live system.

What we would tell anyone doing this

• Give agents a real, signed-in browser, and keep its login across restarts. A signed-out page that looks normal is the most common way to get a confident, wrong report.
• Write down which errors are about the agent’s environment (a tunnel, a missing browser, a hidden window) and which are about the site, and say which conclusions each one supports.
• Make agents check the build actually loaded in the browser before reasoning about what the code should show.
• Test browser tooling in a real browser. Most of our bugs were in the plumbing between the agent and the browser, and mocks hide plumbing.