Yoystro and six models on the same fifty tasks

Summary

The same fifty SWE-bench Verified tasks were run by Yoystro and six models on the same hardware, assessed by the same independent verification. The system under measurement is Yoystro, a cognitive AI infrastructure.

Yoystro completed 48 of 50 tasks (96.0%), spent $0.15 per completed task and finished all fifty tasks in roughly 64 minutes of AI run time. The best comparison model reached 46/50 (92.0%) at $0.33 per completed task; the most expensive model completed 42/50 at $1.07 per completed task. The difference does not come from model strength: the model tier Yoystro uses reaches only 43/50 when run on its own, and the same model reaches 48/50 under Yoystro.

1Introduction

Yoystro is not a coding tool but a cognitive AI infrastructure: an organization's memory of its own codebase, together with an auditable workflow that puts that memory to use.

The word "cognitive" is not a metaphor here; it names the components that were measured: the enterprise memory layer, the rule-based work layer, adaptive routing and verified delivery. Every figure in this report is a joint measurement of those four components; none of them is attributable to the capability of a single model.

The infrastructure addresses three problems. The first is the economics of usage: in continuous work the cost of model calls and the provider side usage allowance become the real constraint on throughput. The second is context continuity: knowledge learned once about a codebase, if it is not carried between sessions, is regenerated and paid for again on every new task. The third is the division of labor: sending work that could be closed in a rule-based way to a model produces both unnecessary cost and unnecessary uncertainty.

The axis of the benchmark follows from those three problems. A completed task rate on its own is not enough. How much the same work costs, how long it takes, and whether knowledge carries across tasks all belong in the same table.

2Method

2.1 Task set and independent verification

The task set consists of 50 real tasks selected with a fixed seed from the 500 records of SWE-bench Verified, frozen for the duration of the runs, so that figures from different days measure the same tasks.

Independent verification is the SWE-bench harness itself, running inside a sealed container with the same image and the same parser for every party. No result is issued by hand. In the final run all fifty tasks received a result; no task went unmeasured.

2.2 Compared parties

On one side is Yoystro's full policy. On the other are six single model command line coding assistants, each running with its own codebase rules file at its default settings. In this published edition those six models are named by class. Each compared tool and model configuration is referred to simply as a "model" in this report.

All six comparison models ran at the base revision, in a clean working copy, with a fifteen minute ceiling per task.

2.3 Fairness principles

The comparison tools were given no hint that would make the work easier: tests were not applied in advance, the success criterion was not disclosed, and no file path was pointed out. The only input each party received is the publicly available issue text of the task itself. The test names independent verification looks for appear in no party's input, and this is guaranteed at container level.

The same task set, the same independent verification, the same hardware and the same price source apply on both sides.

2.4 Cost and time

Cost. For provider 2 models, token consumption was multiplied by published list prices. For provider 1 models, the total cost reported by the tool itself was used, and that figure was additionally cross checked against list prices. No model was recorded at zero cost: zero would have meant "ran for free".

Time. Time means tool and model working time. Independent verification's container run and any waiting on usage allowance appear in no time column.

3Results

3.1 Main comparison

95% confidence intervals (Wilson method) partially overlap on the score axis (Yoystro 86.5% to 98.9%; nearest model 81.2% to 96.8%). With a sample of fifty tasks this is expected, and the report does not hide it. On the cost axis there is no overlap: Yoystro's $0.15 per completed task is less than half of its nearest competitor.

3.2 Difficulty classes

By the official labels the set splits into 22 easy, 20 medium and 8 hard tasks. The hard class is where a party's real capability shows: on easy tasks all parties converge.

The hard class splits the field in two: D completes 8/8, Yoystro and E complete 7/8, while A, B, C and F fall to 4/8, that is, to half. In the easy class Yoystro and D both reach the ceiling at 22/22. Yoystro's overall lead does not come from a single class; it comes from standing at or tied for the top in all three, and it produces its two task margin in the medium class.

3.3 Time and context consumption

One model finishes in less time than Yoystro: F completes 42 tasks in 61 minutes, while Yoystro completes six more tasks in three minutes longer. The slowest model takes more than twice the time to finish six tasks fewer.

The gap on the context side is larger still: Yoystro finishes fifty tasks on 10.8 million tokens of context read, while model B consumes 71.2 million and model A 27.9 million, and both complete six or four tasks fewer. On the output side Yoystro stays at 119 thousand tokens where model B reaches 690 thousand. Not relearning on every task what has been learned once about an organization shows up directly in this column.

3.4 Layers and routing

3.5 Usage allowance efficiency

The number of tasks expected to be completed within a fixed spending window answers the question "how much work gets done on the same budget" directly.

Within the same hundred dollar window Yoystro completes 28% more tasks than its nearest model and roughly 7.3 times more than the most expensive one.

4Advantages

Each of the three headings below rests on a figure measured in section 3. None of them is the achievement of a single model; each is the measured sum of the infrastructure's components.

4.1 Cost: the same work at a quarter to a seventh of the price

At $0.15 per completed task Yoystro is the lowest across the seven-way comparison. The highest scoring comparison model does the same work for $0.33, the most expensive one for $1.07. The total bill for fifty tasks is $7.02 for Yoystro and $44.92 for the most expensive model. This gap is a structure, not a discount: part of the work never reaches a model, and the part that does is closed at the cheapest sufficient level. That the most expensive model tier was never called in this run is the indicator for it.

4.2 Speed: fifty tasks in a little over an hour

All fifty tasks finished in roughly 64 minutes of AI run time. The nearest model in the same score band spent 78 minutes and the third model 84 minutes. The only model that looks faster completes 42 tasks in 61 minutes, while Yoystro completes six more in three minutes longer. The slowest model takes 134 minutes to finish six tasks fewer.

There is a figure on the delivery side as well: total delivery, including queue, preparation and verification, is 138 minutes. Those 138 minutes are therefore not "a change was produced" but "a change was verified".

4.3 Better work: at the top in all three difficulty classes

48/50 is the highest across the seven-way comparison, and the class breakdown shows this does not come from one place: 22/22 in the easy class, 19/20 in the medium class, 7/8 in the hard class. In the hard class one party is one task ahead at 8/8 and one party is level at 7/8; the remaining four fall to 4/8.

The difference does not come from the model's capability, and the indicator sits in the same table: the model tier Yoystro uses reaches only 43/50 when run on its own, and the same model reaches 48/50 under Yoystro. That five task difference was won not by a model upgrade but by how the work is driven.

5Conclusion

On the same fifty tasks Yoystro completes 48/50, spends $0.15 per completed task and finishes the set in roughly 64 minutes of AI run time. The best comparison model completes 46/50 at $0.33 per completed task.

The difference does not come from the model: the model tier Yoystro uses reaches only 43/50 on its own and 48/50 under Yoystro, and two tasks close without reaching a model at all.

Appendix A. Price classes and cost conversion

For provider 2 models, cost is token consumption multiplied by published list prices. Input served from cache is priced separately.

For provider 1 models, cost is the total reported by the tool itself, thinking tokens included. That figure was additionally cross checked against list prices; in two models the conversion came out below what the tool reported and in one model above it, and the table used the tool's own figure in all three cases.

Appendix B. Independent verification reports

The table below carries independent verification's result and test results for each of the fifty tasks. FAIL_TO_PASS are the tests a change must make pass; PASS_TO_PASS are the tests it must not break. The task identifiers are the public SWE-bench Verified identifiers. The table covers the main product run only.

Appendix C. Produced changes

Every change produced for the fifty tasks is reproduced in a separate document, Verification data: independent verification reports and produced changes. That document carries independent verification's summary result and the full change for each task. The raw files (independent verification reports, run logs and changes) are also distributed as a single verification bundle.