SWE-2 benchmarks: results, limits and how to test it
By CodeAgentSwarm · Updated September 24, 2026
What is SWE-2, and how is it related to Devin?
SWE-2 is Cognition’s coding model. Devin CLI is an agent that can run it and other models. SWE-bench is a benchmark name, not another name for SWE-2. Keep the model, agent and evaluation separate when reading a score or choosing what to install.
To try it, follow Devin CLI setup, open /model and select SWE-2 from your account’s catalog. Check the pricing and quota guide before a long test. The published comparison below was checked against Cognition’s source on September 22, 2026.
Results published by Cognition
Cognition published this comparison on September 10, 2026. These are vendor-reported figures, not tests run by CodeAgentSwarm.
| Benchmark | SWE-2 | Fable 5.1 | GPT-6 Astra |
|---|---|---|---|
| FrontierCode 1.1 Main | 50.0% | 50.9% | 53.3% |
| DeepSWE 1.1 | 73.0% | 67.4% | 74.1% |
| Terminal-Bench 2.1 | 92.8% | 91.4% | 89.9% |
| Terminal-Bench 4 | 27.3% | 55.8% | 57.9% |
SWE-2 stands out on Terminal-Bench 2.1 but trails substantially on Terminal-Bench 4. The choice depends on the task.
How to read this comparison
The methodology appendix combines public results with internal evaluations, uses different agents and selects the best reasoning effort per model. It is not a fixed-configuration comparison.
FrontierCode 1.1 Main contains the 100 hardest tasks from Extended. Cognition develops both this benchmark and SWE-2. Keep that relationship in mind when interpreting the results.
Keep the benchmark name and version attached to every score. An aggregate result is not the probability that an agent will solve your next bug: your project may use different tools, languages and constraints.
Choose a specific reasoning effort
The model documentation lists SWE-2 Medium, High and Max. Record which one you test so you can repeat the comparison.
Start with the effort you would normally use and retry failures that justify more time. If you change effort, keep the failed attempt in your record too: discarding it would make the comparison look better than it was.
A small test on your repository
This is our suggested local evaluation; we have not run this protocol as a SWE-2 benchmark. Pick a reproducible bug, an interface change and a maintenance task that you know well.
- Use the same starting commit, dependencies and tests for each attempt.
- Give each model the same request and tools. Keep their changes separate.
- Record total time, human intervention and actual usage, including retries.
- Review the diff and run the tests before scoring the solution.
Task | Model + effort | Tests passed | Review fixes | Elapsed time | Usage
Bug fix | SWE-2 High | ... | ... | ... | ...A useful result is a change you would accept in the project. Count your own fixes and review time; a quick response can become expensive if it needs corrections.
Define success before reading the answer
Fix the failing test described below. Start from the supplied commit.
Preserve the public API and avoid unrelated dependency changes.
Explain the root cause, make the smallest fix and run the relevant tests.
Report commands, results and anything you could not verify.Replace “the failing test” with a real reproduction and expected behavior. Run the project check yourself from the final checkout. Reject a solution that makes the test pass by deleting its assertion or changing the required behavior. Count a manual rescue as intervention even if the final diff is good.
| Record | Why it changes the conclusion |
|---|---|
| First attempt and retries | A successful retry still consumed time and usage. |
| Human review minutes | Correct-looking code can cost more to verify than to generate. |
| Tests that passed and tests not run | A confident final message is not evidence of a working change. |
Keep failures and incomplete runs in the sample; otherwise your success rate describes only the attempts you chose to keep.
Assess cost after checking quality
For Pro pricing, the temporary promotion and its end dates, see the Devin models and quota guide. Record your account terms on the test date so the result remains understandable when the offer changes.
If you have not installed the agent, start with Devin CLI installation and login.
FAQ
Did CodeAgentSwarm run these benchmarks?
No. The table contains results published by Cognition. The suggested local test is a way to evaluate the model on your own project.
Is SWE-2 the best model for every task?
The table does not establish that. Test representative tasks and consider correctness, human review, time and usage before choosing.
Can I compare the CodeAgentSwarm beta with another CLI?
Yes, but you will be comparing the model together with its execution environment. Record the tools, permissions and settings in each environment.