@EpochAIResearch profile picture
Epoch AI @EpochAIResearch

Investigating the trajectory of AI for the benefit of society.

Joined May 2022
  • Posts 2,428
  • Following 0
  • Followers 47,714
840 Photos and videos
Pinned Post
Epoch AI
 @EpochAIResearch
Jun 26
What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.
1:11
11
44
285
64,470
This role is fully remote and we can hire in many countries. Applications are rolling—apply soon! [jobs.lever.co/epoch-ai/d1726…]

Epoch AI - Software Engineer, Benchmarking

Epoch AI is looking for a Software Engineer who will help us evaluate frontier AI models, enabling researchers, developers, and policymakers to better understand AI development. The role will involve...

jobs.lever.co
1
8
2,450
We’re after strong software engineers (2 yrs building complex systems) who bring their own ideas for new benchmarks. AI or cybersecurity experience is a plus, not a must.
1
5
2,580
We’re hiring a Benchmark Engineer to join our Evaluations team! You’ll help expand our AI Benchmarking Hub - running and maintaining benchmarks, integrating with AI providers, and designing brand-new benchmarks from scratch.
3
6
37
5,940
This role is fully remote and we can hire in many countries. Applications are rolling, so apply here now! jobs.lever.co/epoch-ai/42a72…

Epoch AI - Researcher, Evaluations

Epoch AI is looking for a researcher to evaluate frontier AI models on hard-to-grade tasks drawn from real-world scenarios. About the role We’re seeking a Researcher to lead a new effort evaluating...

jobs.lever.co
11
2,167
We are particularly looking for people with research and data-analysis experience who are comfortable working with AI agents. If you already have opinions on what benchmarks do and don't tell us, you might be a great fit.
1
10
2,288
We're looking for new Researchers to join our Evaluations team! Help us curate real-world task suites, design rubrics, and evaluate how well frontier models handle open-ended tasks.
3
9
60
7,143
The open question: can defenders use frontier models to patch vulnerabilities faster than attackers can exploit them? CVE disclosures are a window into how that contest is playing out. Track and explore the CVE data yourself: epoch.ai/data/cve?view=graph

Cyber Vulnerabilities

Explore trends in software and hardware vulnerabilities (CVEs) since 2020 — how counts and severity have changed over time, broken down by the organizations that report them.

epoch.ai
1
60
5,278
Anthropic says Glasswing has surfaced 10k high- or critical-severity vulnerabilities so far (some remain unpublished). OpenAI's Daybreak program likely adds more. The spike in CVEs likely reflects this wave of AI-assisted discovery. Full Data Insight: epoch.ai/data-insights/cve-s…

Disclosed CVEs: 3.5× Spike After Claude Mythos

Notable organizations disclosed ~1,300 high- and critical-severity CVEs in June 2026, roughly 3.5× the pre-Mythos monthly record. Epoch AI's breakdown of vulnerability disclosures following Anthrop...

epoch.ai
2
2
87
7,587
The surge follows Anthropic's April announcement that Claude Mythos Preview could autonomously discover software vulnerabilities, and that trusted partners had been using it to find and fix bugs ahead of the model's public release.
1
3
95
6,752
AI appears to be finding software vulnerabilities at scale. In June 2026, 21 notable organizations disclosed ~1,500 high- and critical-severity CVEs, over 3.5× the previous monthly record set before Claude Mythos Preview's release.
31
104
742
285,357
After Anthropic’s Claude 3 Opus overtook GPT-4 in February 2024, 17 unique models have taken the lead, with the most recent being OpenAI’s GPT-5.5 Pro. The median duration of each lead is about 7 weeks. Full data and methodology in our latest Data Insight: epoch.ai/data-insights/gpt-4…

GPT-4 led in ECI far longer than any other model

GPT-4 topped Epoch's Capabilities Index (ECI) for roughly a year after its March 2023 release — no model since has led for as long. The second-longest reign, by o1, lasted barely three months.

epoch.ai
3
44
4,347
OpenAI’s GPT-4 led the Epoch Capabilities Index for 352 days after its March 2023 release, far longer than any model since. The second-longest lead belongs to OpenAI’s o1 at 98 days.
10
15
316
35,582
Read more about the benchmark on our website. epoch.ai/publications/earthb…

Can AI Learn From Experience? EBR-Bench Results | Epoch AI

Frontier models show no improvement across 30 playthroughs of the board game Earthborne Rangers, scoring far below expert humans. Epoch AI's EBR-bench probes whether AI can learn on the fly.

epoch.ai
2
38
2,812
AI could likely get better at EBR with focused RL training, and we suspect that AI companies have just not prioritized such tasks. So long as this remains the case, EBR-bench serves as a tool to detect the emergence of on-the-fly learning.
1
35
2,813
Have we under-elicited AI’s true capabilities? In the future, we plan to experiment with providing more tools (web search, code execution), trying different scaffolds, using multi-agent setups, and providing expert human playthrough transcripts. Let us know your ideas here!
1
47
2,884
Even if we give them a full strategy guide—the best set of notes we think they could take—models improve only modestly and still show no ability to get better with practice.
1
1
43
3,220
Models also struggle with strategy. A major aspect of this is deck-building, where the player chooses their initial cards. There are 32 “archetypes” of deck but models explore only a fraction of them. Many models stick to a single archetype in all their exploratory playthroughs.
1
33
1,804
Models struggle with tactics. The game’s core damage mechanic is called “fatigue”, and taking too much fatigue is a sign of managing turn-by-turn play poorly. Models do better than random, but fall short of expert human performance.
1
35
1,960
Baseline performance has improved somewhat with newer generations of models. GPT-5.5 and Opus 4.8 clearly outscore GPT-5 and Opus 4.1, though progress since is less obvious. In any case, this comes from better out-of-the-box performance, not from on-the-fly learning.
1
46
9,250
AI systems play the game repeatedly. They are given the rulebook, a card database, and the game’s map. They have a note-taking tool that persists across compactions. Their task is to maximize their score on the final 20% of playthroughs. We see no on-the-fly learning.
1
48
2,704
Load more