I was fascinated to discover yesterday that someone at GiveWell had evaluated the Singularity Institute (SI) as a charity to potentially recommend. Holden Karnofsky, Co-Executive Director of GiveWell, published a post at lesswrong.com (a site [unofficially?] affiliated with SI).

I’ve stopped reading after reading the first ten paragraphs of “Objection 2 …” about what Holden calls “tool-AI”. The basic distinction he makes between tool-AI and agent-AI is that tool-AI is analogous to (or a general category containing) Google Maps – an effective and intelligent tool that does not seem to pose any obvious dangers to anyone because it is neither designed nor allowed to directly affect its environment. Agent-AI on the other hand is explicitly intended to affect its environment.

I think Holden underestimates how prevalent agent-AIs already are, even tho we would likely both agree that they aren’t very intelligent. Newer heating/cooling systems are largely controlled by what is an autonomous agent nowadays; as are elevators and financial trading ‘algorithms’. More and more we are already turning over increasing responsibility to autonomous computing systems that we have designed and built.

I think Holden also misunderstands why we are already, and we will be increasingly, turning over more and more responsibility to these agent-AIs – there is a huge benefit in not requiring a human being to review the decisions of AIs or serve as the proxy agents for these AIs. One reason is that requiring an (intelligent; compared to other humans) human to review the output of a tool-AI is expensive and risky in-and-of-itself – human beings are not designed to perform arbitrary computations efficiently, we perform some types or classes of computations poorly, and we are plagued by various biases and saddled with a number of no-longer-so-appropriate heuristics. These same drawbacks afflict the requirement that humans serve as the proxy agents. In effect, the whole point of AI is too increasingly rely on them as agents – so that human agency can be freed for other pursuits and other responsibilities.

The reason why I believe SI is (rightly) focusing on agent-AI instead of tool-AI is that it is probably inevitable, if it’s at all possible, that someone or (more likely) some group will attempt to develop agent-AI that is generally more intelligent than humans and possibly even capable of improving its own intelligence. Given that, it’s hugely important that these AIs be ‘friendly’ (in the sense of not transforming our world, or the universe, into a living hell).

Tool-AI will continue to be developed – but so too will agent-AI, because we’re already relying on agent-AI to some degree.