Themata.AI
Themata.AI

Popular tags:

#developer-tools#ai-agents#discussion#trending#llms#claude#ai-ethics#code-generation#ai-safety#openai

AI is changing the world. Don't stay behind. Clear summaries, community insight, delivered without the noise. Subscribe to never miss a beat.

Β© 2026 Themata.AI β€’ All Rights Reserved

Archive

|

Topics

|

Privacy

|

Cookies

|

Contact
πŸ•’ LatestπŸ”₯ Top

Filtering by tag:

software-engineeringClear
MirrorCode: What's the largest software project AI can complete on its own? | Epoch AI
code-generationai-benchmarkssoftware-engineeringdeveloper-tools
Research

What's the largest software project AI can complete on its own?

MirrorCode is a benchmark designed to evaluate AI models on long-horizon coding tasks by requiring them to reimplement entire programs end-to-end without access to the original source code. Solutions generated by AI must match the output of the original program to be considered successful.

epoch.ai

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

4 min

8/3/2026

Does code cleanliness affect coding agents? A controlled minimal-pair study

A study investigates the impact of code cleanliness on the performance of autonomous coding agents. The research focuses on how the structural and stylistic quality of code influences an agent's ability to navigate and complete coding tasks.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

7/5/2026

Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

LLM-based Multi-Agent (LLM-MA) systems automate complex software engineering tasks, including requirements engineering, code generation, and testing. Understanding their operational efficiency and resource consumption is crucial for practical adoption.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

6/7/2026

DeepSWE: A contamination-free benchmark for long-horizon coding agents

DeepSWE is a long-horizon software engineering benchmark designed to evaluate coding agents on original engineering tasks. It features contamination-free tasks, high diversity across 91 repositories in five programming languages, and real-world applicability.

deepswe.datacurve.ai

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

20 min

5/26/2026

Speed at the cost of quality: Study of use of Cursor AI in open source projects (2025)

Cursor AI enhances short-term development speed in open-source projects by leveraging large language models (LLMs). However, this acceleration may lead to increased long-term complexity in software maintenance and quality.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

3/16/2026

Many SWE-bench-Passing PRs would not be merged

Approximately 50% of test-passing SWE-bench Verified pull requests created by AI agents between mid-2024 and late-2025 would not be merged into the main branch by repository maintainers. The findings suggest that the lack of iterative feedback for AI agents does not indicate a fundamental capability limitation.

metr.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

18 min

3/11/2026

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

Large language model-powered agents can automate software engineering tasks, including static bug fixing, as shown by benchmarks like SWE-bench. Real-world software development requires navigating complex requirements beyond these capabilities.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

3/8/2026

What's the largest software project AI can complete on its own?

MirrorCode is a benchmark designed to evaluate AI models on long-horizon coding tasks by requiring them to reimplement entire programs end-to-end without access to the original source code. Solutions generated by AI must match the output of the original program to be considered successful.

epoch.ai

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

4 min

8/3/2026

Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

LLM-based Multi-Agent (LLM-MA) systems automate complex software engineering tasks, including requirements engineering, code generation, and testing. Understanding their operational efficiency and resource consumption is crucial for practical adoption.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

6/7/2026

Speed at the cost of quality: Study of use of Cursor AI in open source projects (2025)

Cursor AI enhances short-term development speed in open-source projects by leveraging large language models (LLMs). However, this acceleration may lead to increased long-term complexity in software maintenance and quality.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

3/16/2026

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

Large language model-powered agents can automate software engineering tasks, including static bug fixing, as shown by benchmarks like SWE-bench. Real-world software development requires navigating complex requirements beyond these capabilities.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

3/8/2026

Does code cleanliness affect coding agents? A controlled minimal-pair study

A study investigates the impact of code cleanliness on the performance of autonomous coding agents. The research focuses on how the structural and stylistic quality of code influences an agent's ability to navigate and complete coding tasks.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

7/5/2026

DeepSWE: A contamination-free benchmark for long-horizon coding agents

DeepSWE is a long-horizon software engineering benchmark designed to evaluate coding agents on original engineering tasks. It features contamination-free tasks, high diversity across 91 repositories in five programming languages, and real-world applicability.

deepswe.datacurve.ai

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

20 min

5/26/2026

Many SWE-bench-Passing PRs would not be merged

Approximately 50% of test-passing SWE-bench Verified pull requests created by AI agents between mid-2024 and late-2025 would not be merged into the main branch by repository maintainers. The findings suggest that the lack of iterative feedback for AI agents does not indicate a fundamental capability limitation.

metr.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

18 min

3/11/2026

What's the largest software project AI can complete on its own?

MirrorCode is a benchmark designed to evaluate AI models on long-horizon coding tasks by requiring them to reimplement entire programs end-to-end without access to the original source code. Solutions generated by AI must match the output of the original program to be considered successful.

epoch.ai

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

4 min

8/3/2026

DeepSWE: A contamination-free benchmark for long-horizon coding agents

DeepSWE is a long-horizon software engineering benchmark designed to evaluate coding agents on original engineering tasks. It features contamination-free tasks, high diversity across 91 repositories in five programming languages, and real-world applicability.

deepswe.datacurve.ai

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

20 min

5/26/2026

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

Large language model-powered agents can automate software engineering tasks, including static bug fixing, as shown by benchmarks like SWE-bench. Real-world software development requires navigating complex requirements beyond these capabilities.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

3/8/2026

Does code cleanliness affect coding agents? A controlled minimal-pair study

A study investigates the impact of code cleanliness on the performance of autonomous coding agents. The research focuses on how the structural and stylistic quality of code influences an agent's ability to navigate and complete coding tasks.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

7/5/2026

Speed at the cost of quality: Study of use of Cursor AI in open source projects (2025)

Cursor AI enhances short-term development speed in open-source projects by leveraging large language models (LLMs). However, this acceleration may lead to increased long-term complexity in software maintenance and quality.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

3/16/2026

Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering

LLM-based Multi-Agent (LLM-MA) systems automate complex software engineering tasks, including requirements engineering, code generation, and testing. Understanding their operational efficiency and resource consumption is crucial for practical adoption.

arxiv.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

2 min

6/7/2026

Many SWE-bench-Passing PRs would not be merged

Approximately 50% of test-passing SWE-bench Verified pull requests created by AI agents between mid-2024 and late-2025 would not be merged into the main branch by repository maintainers. The findings suggest that the lack of iterative feedback for AI agents does not indicate a fundamental capability limitation.

metr.org

πŸ”₯πŸ”₯πŸ”₯πŸ”₯πŸ”₯

18 min

3/11/2026

No more articles to load