Read the issue and the code.
Repository-level tasks test more than isolated code generation: the agent must locate the relevant behavior and understand its context.
Quality report / SWE-bench Verified
Coding quality.
Measured in working fixes.
Amber scored higher than GPT-5.6 Sol in an evaluation of 50 tasks from SWE-bench Verified: 88% vs 86%. That’s 44 real repository issues resolved, compared with Sol’s 43.
Internal evaluation · Fixed 50-task subset · Not a full SWE-bench Verified score
01 / Matched evaluation
Amber takes the lead.
Issues resolved on the same 50-task SWE-bench Verified subset. Higher is better.
2BA Amber Extra-high reasoning
88%44 / 50 resolved
GPT-5.6 Sol medium reasoning
86%43 / 50 resolved
Amber leads by two percentage points. One more issue resolved in this evaluation; not evidence of overall superiority.
Amber run: 2 September 2026. Sol run: 28 August 2026. Both submitted all 50 patches; neither run recorded an evaluation error.
02 / What quality means here
Not a convincing answer.
A fix that passes.
SWE-bench uses real issues from open-source repositories. The agent must work with an existing codebase and produce a patch. The evaluation checks whether that patch resolves the issue without breaking the required regression tests.
Produce a working patch.
The model operates through a coding-agent scaffold. Results measure the model, its instructions and its tools together.
Let the tests decide.
“Resolved” is the benchmark harness result, not a subjective rating of how polished the model’s explanation sounds.
03 / Experimental setup
Same tasks.
Same agent scaffold.
We ran both configurations against the same fixed subset. The comparison is between these specific agent configurations, not every possible setting of either model.
- Dataset
- SWE-bench Verified, a human-validated collection of 500 repository issues. This evaluation uses 50 instances, selected with seed 42.
- Agent
- mini-swe-agent 2.4.6, with four rollout workers.
- 2BA configuration
- Amber, extra-high reasoning (
xhigh), self-hosted by 2BA. - Reference configuration
gpt-5.6-sol, medium reasoning, accessed through the Responses API. Model name as recorded in the benchmark.- Scoring
- Resolved instances divided by all 50 selected tasks. One recorded run per configuration; no cross-run union or best-of-N score in the headline.
- Scope
- Task-resolution quality only. These figures do not measure latency, throughput, cost, or general-purpose model capability.
04 / Interpretation & limitations
Close results.
Clear boundaries.
A subset, not a leaderboard claim.
These are internal results on 50 of the 500 Verified tasks, not an official full-suite submission. A one-task difference does not establish statistical equivalence or superiority.
Configuration matters.
Amber’s extra-high reasoning configuration was selected after several prompt and effort experiments on this same subset. It is not an untouched holdout, and that tuning can favor the reported result.
Similar totals, different misses.
Both configurations resolved 38 of the same tasks. Amber alone resolved six; Sol alone resolved five. Similar scores do not mean interchangeable answers on every issue.
Evidence, not a guarantee.
Single runs do not quantify run-to-run variation. Public benchmark exposure may also affect results. Validate the model on your own repositories and review generated patches.
Sources & further reading
The headline uses the completed NVFP4 Amber reference run and the matched Sol run in our benchmark scorecard. Later hardware and incomplete experimental runs are not pooled into this comparison.
05 / Your code is the next test
Put the quality to work.
Try 2BA with your coding agent. Pair your browser, create an API key and configure your tool.