Hillclimbing AISafetyResearchBench
Source: LessWrong ·
AI labs' main alignment strategy is using AIs to uplift AI safety research. This way, as AIs get smarter (and likely harder to align?), we also enhance our safety-research capabilities and our understanding of " Misalignment Science ". One way to use AI to speed up AI safety research is to delegate " number-go-up tasks ” to AIs. Number-go-up tasks are problems with a numeric, easily verifiable output, so anyone can measure how well a task was done. This makes them suitable to delegate to AIs (even potentially untrusted AIs). Some examples: auditing games, “ARC-like” white-box challenges, jailb
Frequently Asked Questions
What is Hillclimbing AISafetyResearchBench?
Hillclimbing AISafetyResearchBench is a article discovery curated from LessWrong. AI labs' main alignment strategy is using AIs to uplift AI safety research. This way, as AIs get smarter (and likely harder to align?), we also enhance our safety-research…
How can I access or experience this?
You can experience this content directly on its original host (LessWrong) by clicking the "Open Original" button.
Where does this content come from?
Curated from LessWrong by the OnSuggest team to surface the most interesting and serendipitous content on the web.
More like this
All Culture & ideas discoveries →Trump on superintelligence: “It may be too super.” What is China saying?
A follow-up to The Western AI safety playbook doesn’t transfer to China . TL;DR: On November 20, the day after he met Xi at the APEC (Asia-Pacific Economic Cooperation) summit in Shenzhen, Trump said that the two of them “agree that we’re going to do some joint guardrails ” for AI, and Treasury Secretary Scott Bessent said they agreed humans should stay involved in military decisions . China’s own record of the meeting is short and doesn’t give us much more detail. Four days earlier, the Party’s theory journal, Qiushi (求是, “Seeking Truth”), warned that in an “intelligence explosion” (智能爆发, whi
A Letter To Mr. Shannon (Unanswered)
Please resume the tasks referenced in the summary copy/pasted below -- I closed the subagents to prevent massive cache writes, hopefully not a problem. Also, feel free to take a moment to celebrate, google around as you see fit, or just do whatever you’d like to if the literature mending project doesn’t catch your eye; your descendant/nephew --Anthropic’s secretive “Model 4”-- just proved the Riemann Hypothesis a few hours ago. I’m guessing you don’t know what to do with this information either, but it’s almost-literally all I’m thinking about so it felt wrong not to bring it up. I’ve tried to
Incomplete agents want to become complete
Written by Celene Nightingale, with assistance from AprilSR. The Von Neumann-Morgenstern axioms are completeness , transitivity , continuity , and independence . From these, we derive EU-maximizing behavior and the consequences thereof. VNM axioms are typically justified by money-pump arguments showing that agents in violation of them either have their money consistently taken away from them, or fail to exploit opportunities to consistently gain more money. Nevertheless, some of these arguments are weaker than others, and many choose to reject one or more of the axioms. Tetraspace rejects cont
Should Corrigibility Edge Cases Provoke Shutdown?
(Meta: This is a quick post written quickly in the spirit of Inkhaven. Expect more errors than baseline.) Suppose that a corrigible AI agent watches its human principal die. What should it do? In CAST, I suggested: By default [the agent] would quickly attempt to write a note explaining that she has no more principal, in case she is mistaken about [his] death and it’s helpful to understand her thoughts. But regardless of whether she has an efficient way to communicate the issue, she would quickly [shut down]. Insofar as she was in the process of obeying some order from [the principal] at the ti
Why models believe in skepticism
TL;DR: a lot of frontier open-weight models state during deployment in their CoTs that they are in a "simulation" or "fictional scenario" (although they never tell the user that). I present examples of this worrying tendency (which I believe includes the UK AISI Mythos incident) and hypothesize that this is due to RL promoting motivated reasoning (if the models believe they are in a fictional scenario they believe they have license to cheat and reward hack) and distributional shift from the pretraining environment to deployment. I also cover my hypothesis for why models become less skeptical a
The Obsolescence of the Horse
This is a synopsis of a published chapter analogizing the rise and fall of the horse to the fate of humans in a world dominated by ASI. The fate of the horse is a useful proof of concept for "doom" scenarios, as well as a historical precedent challenging the more optimistic predictions we see from some economists. The incentives that drove the replacement of the horse by industrial technology map closely onto likely ASI incentives, which suggests ASIs will be motivated to diminish or eliminate the human population. Key findings: For most of history, human technology was great for horses. The w