The hardest roles to fill in the AI safety ecosystem seem to be: - Senior executives (CEO, COO, etc.) - Research leads (Chief Scientist, Scientific Director, etc.) - Program Manager or other roles for "highly competent generalists with strong AI safety knowledge and connections"

New from Scott Alexander: "The AI Superforecasters Are Here" astralcodexten.com/p/the-ai-…

Lots of ways that the default path of AI development marches towards power concentration, and a lot of possible solutions are pretty mixed. For example - the US gov gets 5% stake in AI companies, more economic redistribution leverage but bad incentive overlap between the state and super-powerful AI companies. Or, broadening access to the frontier is v good for avoiding domination but could give bad actors the ability to consolidate power, etc. Tons of thinking and work to do here and excited to be supporting it through this RFP with the incredible EIP CIP teams, esp Ashwin, @tyler_m_john, and @JoalStein. Can’t believe it’s taken me this many years in AI to co-author a beige microsite. More soon!

4️⃣ Experts and superforecasters give a 50% chance that a major government restricts an AI release by 2030 We asked for the year by which the U.S., UK, or EU governments would delay, seriously restrict, or condition the public release of an AI system on safety grounds. Experts and superforecasters gave a median prediction of 2030, while the public predicted a slightly later date, 2032. The current export-control directive suspending access to Anthropic's Mythos 5 and Fable 5 models doesn't meet our resolution criteria, as these models were already publicly available before the directive was issued. We will monitor news surrounding GPT-5.6 and related models to see if near-term government actions meet the criteria.

Today we're launching Intercept: a $500M philanthropic initiative to make respiratory infections, like the common cold and flu, a thing of the past. We treat respiratory infections as a minor nuisance, but that’s really not the case. Most of us will spend 5% of our lives (!) sick from these viruses, they kill 1M people a year, cost $600B annually in productivity, and periodically threaten civilization through pandemics. So, if they’re such a big problem, why haven’t we dealt with them yet? Last year we convened ~40 leading scientists, pharma R&D leaders, biotech investors, and regulatory experts to better understand that. We heard two main reasons: (1) First, it’s just technically very challenging: respiratory viruses represent hundreds of distinct, mutating strains across several families. Fortunately, recent breakthroughs make this newly possible. (2) Second is a lack of funding: broad-spectrum solutions have historically been underfunded, in part because they’re not a great fit for most philanthropic or commercial funding (and while COVID generated a burst of activity around preventing and understanding respiratory infections through an influx of new funding, that hasn't been sustained). We think that with enough focus and funding, this might be solvable. Intercept is a $500 million philanthropic initiative that will take advantage of new tools to catalyze the development and deployment of two types of products: broad-spectrum preventatives and air cleaning technologies. This problem is undoubtedly difficult. But it’s more tractable now than it’s ever been. We think we should give it our best shot. We’re enormously grateful to our anchor funders: @stripe, @AnthropicAI, @TheFluLab, @FoundationOAI and individuals from Jane Street. And, I’m very excited to be building this with @incredutility and the rest of the team.

I am pleased and honored to announce that, on July 6, I'll be joining @OpenAI as leader of a new team called Strategic Futures. Our mandate will be to help the company's leadership shape frontier AI policy. There is a ton of work to do, and I'm excited to get started.

The reason frontier AI companies are struggling to find the right balance between biosec and usefulness is because there aren’t enough competent bio ppl in these rooms. Longterm that will hurt biosec efforts.

In medieval times, within the arms race of ever more demonic torture devices, some sadistic genius came up with the idea of the Little Ease. This was a prison cell built so small in every dimension that a grown man could not stand upright in it nor lie down at full length nor properly sit. The pain is relentless and without relief and inflicted by one's own body. Prisoners were known to go insane within a few days. A stay at the Little Ease was considered even more cruel than the rack, the thumbscrew, and the other ghoulish machinery of the Tower of London. A breeding pig will spend her whole life in a version of that box. These are social, roaming creatures (more intelligent than dogs) who will never leave this corset of steel. They have been selectively bred to be bigger than their frames can support. Yet we put them in cells so confined that they cannot comfortably sit, and their attempts to do so (for example, by sneaking their limbs into adjacent stalls) reliably lead to fractures and sprains. They cannot sweat, yet have nothing to roll around in to cool themselves off. Except their own manure, which (contrary to the common misconception) they are so averse to (thanks to their strong sense of smell) that new sows will often suffer from constipation to avoid soiling the space from which they eat and sleep. Here is how the writer Matthew Scully described what saw at one of Smithfield’s “gestation barn”: > “Sores, tumors, ulcers, pus pockets, lesions, cysts, bruises, torn ears, swollen legs everywhere. Roaring, groaning, tail biting, fighting, and other “Vices,” as they’re called in the industry. Frenzied chewing on bars and chains, stereotypical “vacuum” chewing on nothing at all, stereotypical rooting and nest building with imaginary straw. And “social defeat,” lots of it, in every third or fourth stall some completely broken being you know is alive only because she blinks and stares up at you … creatures beyond the power of pity to help or indifference to make more miserable, dead to the world except as heaps of flesh into which the [insemination] rod may be stuck once more and more flesh reproduced.” — The Save Our Bacon Act is trying to unroll the few state protections we have against this barbaric cruelty - for example California’s Prop 12 - which banned the sale of pork from pigs kept in gestation crates. It’s incredibly important we don’t end up with this sort of federal preemption. SOB will not only kill the most important animal welfare related laws in the US of the past decade, but more importantly, it will also restrict ALL future legislative progress (aka how the animal welfare movement has gotten its biggest wins). The Senate is currently deciding whether to add the SOB Act to the Farm Bill. With relatively little money now, we can discourage the most pivotal senators in the Ag committee from backing this amendment. Defeating this bill is even more important given the amount of philanthropic funding I expect to come online in the next year or two. It will plausibly be over 10x more expensive to repeal SOB than to prevent it from passing in the first place. All that money that could be spent transforming our society's relationship to mass animal suffering will instead have to be spent just getting us back to where we are right now. That's why money spent now fighting this bill (and I mean right NOW) is so effective. If you’re in a position to donate six figures, please DM me.

Andrew, I signed, this is important work - but we need to be really careful to not get distracted from the bigger problem of rapidly decentralizing capabilities to create / deploy biothreats. Detection, identification, response - globally and persistent/pervasive - should be the top priority. DNA synth will be distributed, it won’t be centralized in a way that can be effective with this approach. Desktop, at home, new synthesis tech coming, etc. Not to mention we can’t regulate China, etc. and this is a globally distributed technology. This kind of legislation is sort of like telling people not to write viruses on all their home computers or, worse, if they do we will monitor them all to see what they type (which people would get around of course). Important now, not practical in medium/long run. Necessary to reduce some risk, not nearly sufficient… nor where our top priorities should be. Only practical thing is to make detection and response work great, global, and fast. e.g. - The "finger in the dike" refers to the classic legend of the "Little Dutch Boy" who selflessly plugs a small leak in a dam with his finger, holding back the sea to save his town from a devastating flood. Small, temporary stopgap to a major, escalating problem.

Sam Altman, Dario Amodei, Demis Hassabis and many others have signed a letter urging Congress to increase security on orders of synthetic nucleic acids - and the equipment needed to make them - as models continue to become increasingly bio-capable.

It’s great to see AI leaders like Sam Altman, Dario Amodei, and Demis Hassabis calling for mandatory DNA synthesis screening, which is a no-brainer policy for preventing (AI-enabled) bioterrorism. But fewer than 50 people in the world currently work on DNA security full-time. We need a comprehensive plan and at least 5x as many people to secure the DNA supply chain before AI and biotech outpace us. @jtmonrad and I spent the past two years developing a field strategy for how to do it. Successfully defending against this risk (while still capturing innovation benefits) requires four things: 1. Coverage: More than 80% of synthetic DNA providers screen both orders and customers 2. Strategic ambiguity: a bad actor can’t easily tell which providers will screen their order 3. Access: legitimate customers can still order DNA cheaply and easily 4. Effectiveness: 90% of providers reliably catch dangerous sequences when red-teamed We’re already seeing real momentum. Many DNA providers screen voluntarily, and governments in several countries are moving toward mandates. But that doesn’t mean the problem will be solved in time by default. Our guide lays out exactly which projects we need to launch. We’re looking for founders, operators, and technical experts to own pieces of the solution. We’re also hiring a Senior Program Officer at Sentinel to drive this work. Get in touch if you or someone you know would be a strong fit! (links for EOI form and JD below) Read our full field strategy in @IFP's Launch Sequence: ifp.org/how-to-secure-the-dn…

No one should be able to order a bioweapon through the mail. @IFP & @JoinFAI are proud to co-lead an open letter calling for mandatory DNA synthesis screening & recordkeeping. Signatories include: - Sam Altman, CEO & Co-Founder, OpenAI - Dario Amodei, CEO & Co-Founder, Anthropic - David Baker, Director, Institute for Protein Design; 2024 Nobel Prize in Chemistry recipient - Patrick Collison, CEO & Co-Founder, Stripe - Paul Graham, Founder, Y Combinator - Demis Hassabis, CEO, Google DeepMind; 2024 Nobel Prize in Chemistry recipient - Emily Leproust, CEO & Co-Founder, Twist Bioscience - Lawrence Lessig, Roy L. Furman Professor of Law and Leadership, Harvard Law School - Gerald W. Parker, former Special Assistant to the President for Biosecurity and Pandemic Response - Mustafa Suleyman, CEO, Microsoft AI - Alex Tabarrok, Professor of Economics, George Mason University - Alexandr Wang, Chief AI Officer, Meta; Founder, Scale AI - Christine E. Wormuth, President & CEO, Nuclear Threat Initiative; 25th Secretary of the Army Read the letter and see the full list of signatories: screendna.org Many DNA synthesis companies voluntarily screen orders to mitigate biosecurity risks, but no law requires them to do so. Leaders in AI, biotech, life sciences, national security, and the nucleic acid synthesis industry agree that Congress should act to strengthen safeguards against biological threats. @deanwball put it well in the WSJ: “If you’re synthesizing the stuff that yields biological life and viruses, we’re asking you to screen to see whether it is dangerous in some way. That seems like a reasonable thing for society to insist upon.”