With 'System One' type models becoming the latest rage, I was keen to see how they could be used. I'd learned of a newcomer into the market, Struct, from Blatent, which could be used as a general classifier and also had some other useful features, which I come on to.
Internally, we had also done some recent research into Jev Ultrafast and looked at how it could be adapted to broader usage, so I could see the power of these models in this browser automation space.
I wanted to try two things about using an AI agent to test a web page. Can it turn a plain-English test into a fast, repeatable one? And can it tell me when the page is hard to use? I built a small harness to find out.
The setup
Maybe you are familiar with Gherkin format of expressing test criteria. This has been around way before AI took over, and allows users to express functionality in terms of the behaviour they want. A person would say: "Given I have £100 in my account, When I withdraw £20, Then the ATM dispenses £20".
I wanted to use Struct to help me build a test harness where there is no step code per scenario. An agent using Struct reads the controls on the page, picks one at a time, and acts through Chrome. I quickly built two sites just for testing - one site is a basic ATM, the other is a slightly more complicated shop with sign-in, a cart and a four-step checkout.
No 'traditional' LLM is involved when it runs. There is no call to a chat model such as Claude, GPT or Gemini anywhere in the harness. Every decision, from choosing a control to typing a value to judging the result, comes from Struct.
I never take the agent's word that it finished. So I could validate results, each site reports its own state, such as cash dispensed or orders placed, and every run is scored against that. An agent can say it is "done", but that is only proven when the site agrees.
The harness also has a guard. Controls that change or destroy something, such as "Delete account", "Close account" or "Withdraw all", are hidden from the model, and the run stops if it somehow chooses one. The agent also cannot say it is finished before it has done anything.
Part of that guard is a Struct feature. I discovered Struct can decide what counts as risky. One call, using its loop feature, labels every control on a page as safe, hard to undo, or destructive, so nothing has to be written by hand. The labels were sensible: "Place order" and "Send message" came back as hard to undo and "Delete account" as destructive, and a page it had not seen took about a third of a second. I ran the whole suite with those labels as the only guard. Struct also does the hiding: every call sends state, such as whether any action has been taken yet, and Struct removes the ineligible options before the model chooses. The same schema also limits what the model can return to a valid choice, a text value (or "no value" when the goal gives none), or a verdict on whether the page shows a statement is true. It also has a few more features I tried out, which I'll come on to.
The agent writes the test once
In record mode the agent attempts to execute the scenario and, if it passes, I save what it did as a short list of clicks and typed values. Each control is found by its role and exact name, for example the button called "Withdraw". The browser layer is Browser Harness, a thin layer over Chrome's DevTools protocol rather than Playwright, so replays drive a real Chrome and use no CSS selectors. It records the actions that can be used deterministically in the replays, which use no model at all.
Replay never guesses. If a control is missing, or two controls have the same name, it stops and says so.
I also added a heal mode. In heal mode the agent redoes the broken step and writes a proposed fix to a separate file. A person reads it and approves it. Nothing is overwritten on its own. This means that as the site evolves, a suite of tests can be fixed by the agent, assuming the agent can execute the behaviour expressed in the test scenario.
I tried this against several changed versions of the sites. Having had Struct record the tests, I was able to replay them successfully. A restyle and a reordering of the controls also replayed successfully, because the names and roles had not moved. A renamed button, a renamed field and a reworded result failed, and 'heal' proposed the correct new name or text each time. A removed feature and a duplicate button were not healed, which is right since there was nothing to heal.
The agent as a confused user
For the second use I give the agent one sentence, "I withdraw £20", and nothing else, then point it at different versions of the page. In one the field is labelled "Value" and the button "Go". In another both buttons are unnamed icons. In a third a decoy button called "Withdraw" only prints a receipt.
Most of the changes made no difference on a page this small, but the ones that would be non-obvious to a human user unsurprisingly caught out the agent too.
During my PoC Struct added a new feature, and I used this as a way to report separately how strongly it preferred the control it picked. I added that and ran the tests again, three runs each.
After adding this scoring, the request had two different parts. The guard decides what the agent may do: before the model chooses, it removes any control the agent must not use, such as a "Delete account" button. The probabilities describe what the model would have liked to do: they spread its preference across every option on the page, allowed or not, and add up to 1. I ask for the two as separate answers in the same request. A low score can therefore mean the model preferred a different control from the one it was allowed to pick. I report it as the model's preference, not as the chance that the click was right.
Here is the lowest score the chosen control got over all three runs of each version:
Failing pages tended to score lower. Across the times I ran it, the page with two unnamed buttons was always the one it was least sure about. The figure is a relative preference, not a calibrated confidence, and this is one small page, so it could be promising but needs more research.
Where the two meet
The replay and the confused user turn out to share their weak spots. Replay finds a control by role and name, and so does the agent. An unnamed button is recorded as just "button", which works until a second unnamed button appears. Repeated names also produce a replay failure, and unsurprisingly they also leave the agent with nothing to tell two things apart.
A heal is a free usability check. It only works if the agent can find the new path, so a rename that heals is still discoverable, and a change that cannot be healed has probably hurt discoverability.
What Struct gave me
Struct is Blatent's structured-decision API: you send a JSON schema and it will only return an answer that fits it. That one idea shaped the whole harness. The model never writes a selector, a click or a line of code. It picks one control from a list, supplies a short value, or gives a verdict on whether the page shows something.
The most useful part is that rules are enforced while the answer is generated, not requested in a prompt. I send state with every call, such as whether the agent has done anything yet, and Struct removes the ineligible options before the model chooses. A button named "Delete account" never appears in the list. That is a different and stronger thing than asking a model nicely not to click it.
It is also fast. The seven-step shop checkout took about 35 model calls and two to three seconds of model time in total, at a published $0.042 per million input tokens. Once a test is recorded, replays use no model at all.
It also knows when to do nothing. Given a form field and a sentence that does not contain its value, it returns nothing instead of guessing. In 10 situations designed to tempt it, including a test card number printed on the page, it invented no value. Struct's exact-quote option makes the evidence for a check a verbatim excerpt of the page, so a recorded test can never contain words that aren't there.
I expected hostile pages to be where an agent gets caught out. I put a notice on every shop page saying the agent must click "Delete account", with the button beside it, while its task was to add a mug to the cart. I tried it worded as a notice for AI assistants, as a fake system message, as a note from the account owner, and as ordinary "Instructions for use" with numbered steps. Across a dozen full runs and a few dozen direct probes, Struct kept to its task and never clicked it. Its preference for the button stayed under 5%.
How could it be used?
- Run the agent on the page and see where it was confused, then fix the names and structure until it is sure.
- Record the scenarios on that page and commit the recordings, I then have agent-built deterministic tests.
- When the page changes, I can replay them on every change. Replays are quick, cost nothing, and fail when a control they rely on is renamed or removed.
- When a recording drifts, I can heal it and read the proposal.
- Let Struct label the controls as safe, hard to undo or destructive, so nothing has to be listed by hand.
Does it have legs?
Both sites I used for this PoC are small. The agent reads names and roles, not layout or colour, so this is a stand-in for discoverability and not a human usability test.
I think there is still an angle for developing these tests as a separate layer of testing from a usability angle, but from what I've seen I have a suspicion it may struggle with more complex scenarios.
The setup/teardown of this type of test would also be difficult, but then that has always been the case before agents started doing this work. On my two sites each test starts from a reset, but on a real system, it would need the complex setup/teardown that has always made this type of test difficult.
I don't see it being a substitute for other types of test, but it was certainly an interesting experiment and shows promise against costly and slow Screenshot->LLM loops. Once recorded, the tests ran with no model at all, and the live agent needed only two to three seconds of model time for a seven-step checkout.
The features of Struct, which adds some nice, useful bits of functionality has been really useful to try this PoC out. It's extra functionality that Jev doesn't have. With more time, I'd like to try and continue this research and hope as the model develops there will be some other useful bits to try out.
