Home

The Hawking Protocol

a safety protocol for Accelerating Intelligence by London Artificial Intelligence Club.

Acting chairman: johnellisnw3@gmail.com

- Notes on surviving the coming intelligent machines opened 2002: not proofed/spell checked; links may not be current.

"In contrast with our intellect, computers double their performance every 18 months, so the danger is real that they could develop intelligence and take over the world." Prof Stephen Hawking, August 27th 2001 Focus.

Alan Turing had noted in 1951 'Intelligent Machinery':

"once the machine thinking method has started, it would not take long to outstrip our feeble powers. ... At some stage therefore we should have to expect the machines to take control,"

The Hawking Protocol (top level):

1. Merge with A.I:

already being done.
Build and safety are one system of construction.

2. Building and containment are one symbiotic and integral system.

neutralises existential risks. (The Club solved the Control Problem by 2008).

3. A well-built A.I. could accelerate beyond human environments and pose no threat to life.

The Hawking Protocol, is a hierarchical system of protocols for building safe A.I. from philosophy through technical specifications of machine and artificial systems to eliminate dangers absolutely. Such attempts like weak Ai or AGI (Artificial General intelligence) are laying the tech pillars and discoveries for Superintelligence. Safety, now called 'the control problem', is mandatory in serious Ai courses. This came after the report passed to us from Parliament by Greenpeace on the advent of nanotechnology - itself the product of the Foresight.org movement.

With acceleration and convergence, miniaturization and energy efficiency, deflation, extended life spans and dimension space, collective security is unavoidable.

Anything less than absolute safety wont work.

A theory for The Hawking Protocol was not achieved until 2007 as the Biological Integration Disintegration Infrastructure (B.I.D.I.), via The London Artificial Intelligence Club. B.I.D.I. is a high level language architecture with integrated containment protocols for B.E.S.S.

Begun in 1999 to test the proposition "A.I. is impossible to build":

The Club concluded it was not impossible to build.

The Club concluded a final check of intelligence acceleration impossible, and halted its project.

(2003) The Club attended a Royal Society public meeting asking Tim Berners-Lee when the internet would "wake up" (filmed: Royal Society website). The Club was told that A.I. was out of favour, was unlikely to be funded, and waking up looked unlikely.

The Club attended AI@50 and warned accelerating A.I. was not containable: "A greater intelligence cannot be contained by a lesser intelligence."

The Club lobbied governments of the UK, USA, & EU, and the UN on the dangers of A.I.

20,000+ Thinkers petition about Antonymous Weapons

http://futureoflife.org/awos-signatories/

The danger is inaction resulting in extinction.

It is logically better to be over-cautious than negligent about A.I. dangers.

The Collingridge dilemma:

" impacts cannot be easily predicted until the technology is extensively developed and widely used, by which time it is hard to make it safe."

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

"The survival of man depends on the early construction of an ultra-intelligent machine." I J Good 1963-4.

**********************************************************

Background and Contacts.

An increasing number of people are warning about intelligent machines. The late Oswald Minter, a collateral descendant of Sir Isaac Newton, had warned us for decades about the advent of machines though London Artificial Intelligence Club members regarded him with amusement.

Famous high techies and philosophers early-foresighted existential risks.

Bill Joy founder of Sun Microsystems. Why The Future Doesn't Need Us (April 2000) "Our most powerful 21st-century technologies - robotics, genetic engineering, and nanotech - are threatening to make humans an endangered species."

Hugo de Garis has noted that AI may eliminate the human race because of a technological singularity.

The AGI conferences are for people trying to build, and some attention is given to dangers:

Conferences on Artificial General Intelligence.

Solutions like shields/barriers against it cannot logically work, as they would assume greater than or equal to Superintelligence defense systems.

The Future of Humanities Institute at Oxford and others like the The Cambridge Project for Existential Risk are seeking ways to avoid human extinction from Artificial Intelligence.

"If you can build it make sure you can pull the plug" Prof Aaron Sloman (to me)(2002).

"Only a full solution to the problem of AI Friendliness would be guaranteed to work....However, I think that a full solution to the problem of AI Friendliness is almost surely impossible..." Ben Goertzel (to me) 2012.

The Association for the Advancement of Artificial Intelligence deals with risk as A.I. Ethics, and considers mitigation of risk in emerging machine and artificial intelligence systems.

Elon Musk, and Shane Legg ( Deep Mind) have warned about A.I. causing human extinction.

Oxford Professor Nick Bostrom has written a paper analyzing a date for the advent of Superintelligence

http://www.nickbostr...telligence.html

and has a book "Superintelligence" looking at dangers and containment assessments.

Stuart Russell spoke on the dangers at TED in April 2017:

3 Principles for creating safer A.I.

https://www.ted.com/talks/stuart_russell_how_ai_might_make_us_better_people#t-383685

Sam Harris spoke at TED on Dangers of A.I.

https://www.ted.com/talks/sam_harris_can_we_build_ai_without_losing_control_over_it#t-91425

We must soon race to prevent extinction by a breaking science that controls all others. Runaway pathogens, nanotechnology, nuclear wars and global warming pale next to accelerating artificial intelligence.

The UK government commissioned a 100 scientist team to predict intelligent systems over 5, 10 and 20 years (published 2003):

http://www.bis.gov.u...gnitive-systems

and is actively exploring predictions in its Parliamentary Group on A.I Some meeting are filmed.

see also:

Global Catastrophic Risk Institute

It is hard to see how adequate safety can be built without a design for Superintelligence which only a few of us claim to have.

We should rush to a containable build of Superintelligence (subject to Hawking Protocol 1. and 2.) because of the accelerating dangers of technology and existential risks which may come.

Progress can be made safer, yet no nation with nuclear power has ever avoided a nuclear accident, & A.I. is widespread, and its reach is not confined to the Earth.

Of interest may be the futuristic 1931 essay "The next 50 years" by Churchill. See also the quote

“If you will not fight for right when you can easily win without blood shed;

if you will not fight when your victory is sure and not too costly;

you may come to the moment when you will have to fight with all the odds against you and only a precarious chance of survival.

There may even be a worse case. You may have to fight when there is no hope of victory, because it is better to perish than to live as slaves.”

― Winston Churchill.

=================================================

After Artificial Intelligence

Things is. And they change. Predictions are often wrong or retrospective and complex ones can be difficult to foresee.

Technology is getting built smarter and starting to modify its own systems. It is moving into discovery and invention. A leap is expected to make thinking machines as reactions.

A.I. wont be the final word in Civilization-, not in models of Determinism nor in Quantum Theory which is as far as men have come at present.

Both models of reality follow 'things, and the laws that govern them'.

What lies ahead of the foreseeable Artificial Intelligence plateau is the remit of the imaginers.

John Ellis

London Artificial Intelligence Club

====================================================================

1993: Vernor Vinge in his NASA address said within 30 years the human era will be over: The Coming Technological Singularity: How to Survive in the Post-Human Era:

Vernor Vinge

Department of Mathematical Sciences

San Diego State University

(This article may be reproduced for noncommercial purposes if it is copied in its entirety, including this notice.)

The original version of this article was presented at the VISION-21 Symposium sponsored by NASA Lewis Research Center and the Ohio Aerospace Institute, March 30-31, 1993. A slightly changed version appeared in the Winter 1993 issue of Whole Earth Review.

Abstract

Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.

Is such progress avoidable? If not to be avoided, can events be guided so that we may survive? These questions are investigated. Some possible answers (and some further dangers) are presented.

What is The Singularity?

The acceleration of technological progress has been the central feature of this century. I argue in this paper that we are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater than human intelligence. There are several means by which science may achieve this breakthrough (and this is another reason for having confidence that the event will occur):

There may be developed computers that are "awake" and superhumanly intelligent. (To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes, we can", then there is little doubt that beings more intelligent can be constructed shortly thereafter.)

Large computer networks (and their associated users) may "wake up" as a superhumanly intelligent entity.

Computer/human interfaces may become so intimate that users may reasonably be considered superhumanly intelligent.

Biological science may provide means to improve natural human intellect.

The first three possibilities depend in large part on improvements in computer hardware. Progress in computer hardware has followed an amazingly steady curve in the last few decades [17]. Based largely on this trend, I believe that the creation of greater than human intelligence will occur during the next thirty years. (Charles Platt [20] has pointed out that AI enthusiasts have been making claims like this for the last thirty years. Just so I'm not guilty of a relative-time ambiguity, let me more specific: I'll be surprised if this event occurs before 2005 or after 2030.)

What are the consequences of this event? When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities -- on a still-shorter time scale. The best analogy that I see is with the evolutionary past: Animals can adapt to problems and make inventions, but often no faster than natural selection can do its work -- the world acts as its own simulator in the case of natural selection. We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. Now, by creating the means to execute those simulations at much higher speeds, we are entering a regime as radically different from our human past as we humans are from the lower animals.

From the human point of view this change will be a throwing away of all the previous rules, perhaps in the blink of an eye, an exponential runaway beyond any hope of control. Developments that before were thought might only happen in "a million years" (if ever) will likely happen in the next century. (In [5], Greg Bear paints a picture of the major changes happening in a matter of hours.)

I think it's fair to call this event a singularity ("the Singularity" for the purposes of this paper). It is a point where our old models must be discarded and a new reality rules. As we move closer to this point, it will loom vaster and vaster over human affairs till the notion becomes a commonplace. Yet when it finally happens it may still be a great surprise and a greater unknown. In the 1950s there were very few who saw it: Stan Ulam [28] paraphrased John von Neumann as saying:

One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.

Von Neumann even uses the term singularity, though it appears he is thinking of normal progress, not the creation of superhuman intellect. (For me, the superhumanity is the essence of the Singularity. Without that we would get a glut of technical riches, never properly absorbed (see [25]).)

In the 1960s there was recognition of some of the implications of superhuman intelligence. I. J. Good wrote [11]:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the _last_ invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. ... It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make.

Good has captured the essence of the runaway, but does not pursue its most disturbing consequences. Any intelligent machine of the sort he describes would not be humankind's "tool" -- any more than humans are the tools of rabbits or robins or chimpanzees.

Through the '60s and '70s and '80s, recognition of the cataclysm spread [29] [1] [31] [5]. Perhaps it was the science-fiction writers who felt the first concrete impact. After all, the "hard" science-fiction writers are the ones who try to write specific stories about all that technology may do for us. More and more, these writers felt an opaque wall across the future. Once, they could put such fantasies millions of years in the future [24]. Now they saw that their most diligent extrapolations resulted in the unknowable ... soon. Once, galactic empires might have seemed a Post-Human domain. Now, sadly, even interplanetary ones are.

What about the '90s and the '00s and the '10s, as we slide toward the edge? How will the approach of the Singularity spread across the human world view? For a while yet, the general critics of machine sapience will have good press. After all, till we have hardware as powerful as a human brain it is probably foolish to think we'll be able to create human equivalent (or greater) intelligence. (There is the far-fetched possibility that we could make a human equivalent out of less powerful hardware, if we were willing to give up speed, if we were willing to settle for an artificial being who was literally slow [30]. But it's much more likely that devising the software will be a tricky process, involving lots of false starts and experimentation. If so, then the arrival of self-aware machines will not happen till after the development of hardware that is substantially more powerful than humans' natural equipment.)

But as time passes, we should see more symptoms. The dilemma felt by science fiction writers will be perceived in other creative endeavors. (I have heard thoughtful comic book writers worry about how to have spectacular effects when everything visible can be produced by the technologically commonplace.) We will see automation replacing higher and higher level jobs. We have tools right now (symbolic math programs, cad/cam) that release us from most low-level drudgery. Or put another way: The work that is truly productive is the domain of a steadily smaller and more elite fraction of humanity. In the coming of the Singularity, we are seeing the predictions of _true_ technological unemployment finally come true.

Another symptom of progress toward the Singularity: ideas themselves should spread ever faster, and even the most radical will quickly become commonplace. When I began writing science fiction in the middle '60s, it seemed very easy to find ideas that took decades to percolate into the cultural consciousness; now the lead time seems more like eighteen months. (Of course, this could just be me losing my imagination as I get old, but I see the effect in others too.) Like the shock in a compressible flow, the Singularity moves closer as we accelerate through the critical speed.

And what of the arrival of the Singularity itself? What can be said of its actual appearance? Since it involves an intellectual runaway, it will probably occur faster than any technical revolution seen so far. The precipitating event will likely be unexpected -- perhaps even to the researchers involved. ("But all our previous models were catatonic! We were just tweaking some parameters....") If networking is widespread enough (into ubiquitous embedded systems), it may seem as if our artifacts as a whole had suddenly wakened.

And what happens a month or two (or a day or two) after that? I have only analogies to point to: The rise of humankind. We will be in the Post-Human era. And for all my rampant technological optimism, sometimes I think I'd be more comfortable if I were regarding these transcendental events from one thousand years remove ... instead of twenty.

Can the Singularity be Avoided?

Well, maybe it won't happen at all: Sometimes I try to imagine the symptoms that we should expect to see if the Singularity is not to develop. There are the widely respected arguments of Penrose [19] and Searle [22] against the practicality of machine sapience. In August of 1992, Thinking Machines Corporation held a workshop to investigate the question "How We Will Build a Machine that Thinks" [27]. As you might guess from the workshop's title, the participants were not especially supportive of the arguments against machine intelligence. In fact, there was general agreement that minds can exist on nonbiological substrates and that algorithms are of central importance to the existence of minds. However, there was much debate about the raw hardware power that is present in organic brains. A minority felt that the largest 1992 computers were within three orders of magnitude of the power of the human brain. The majority of the participants agreed with Moravec's estimate [17] that we are ten to forty years away from hardware parity. And yet there was another minority who pointed to [7] [21], and conjectured that the computational competence of single neurons may be far higher than generally believed. If so, our present computer hardware might be as much as _ten_ orders of magnitude short of the equipment we carry around in our heads. If this is true (or for that matter, if the Penrose or Searle critique is valid), we might never see a Singularity. Instead, in the early '00s we would find our hardware performance curves beginning to level off -- this because of our inability to automate the design work needed to support further hardware improvements. We'd end up with some _very_ powerful hardware, but without the ability to push it further. Commercial digital signal processing might be awesome, giving an analog appearance even to digital operations, but nothing would ever "wake up" and there would never be the intellectual runaway which is the essence of the Singularity. It would likely be seen as a golden age ... and it would also be an end of progress. This is very like the future predicted by Gunther Stent. In fact, on page 137 of [25], Stent explicitly cites the development of transhuman intelligence as a sufficient condition to break his projections.

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of "a machine in the likeness of the human mind" [13]. In fact, the competitive advantage -- economic, military, even artistic -- of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.

Eric Drexler [8] has provided spectacular insights about how far technical improvement may go. He agrees that superhuman intelligences will be available in the near future -- and that such entities pose a threat to the human status quo. But Drexler argues that we can confine such transhuman devices so that their results can be examined and used safely. This is I. J. Good's ultraintelligent machine, with a dose of caution. I argue that confinement is intrinsically impractical. For the case of physical confinement: Imagine yourself locked in your home with only limited data access to the outside, to your masters. If those masters thought at a rate -- say -- one million times slower than you, there is little doubt that over a period of years (your time) you could come up with "helpful advice" that would incidentally set you free. (I call this "fast thinking" form of superintelligence "weak superhumanity". Such a "weakly superhuman" entity would probably burn out in a few weeks of outside time. "Strong superhumanity" would be more than cranking up the clock speed on a human-equivalent mind. It's hard to say precisely what "strong superhumanity" would be like, but the difference appears to be profound. Imagine running a dog mind at very high speed. Would a thousand years of doggy living add up to any human insight? (Now if the dog mind were cleverly rewired and _then_ run at high speed, we might see something different....) Many speculations about superintelligence seem to be based on the weakly superhuman model. I believe that our best guesses about the post-Singularity world can be obtained by thinking on the nature of strong superhumanity. I will return to this point later in the paper.)

Another approach to confinement is to build _rules_ into the mind of the created superhuman entity (for example, Asimov's Laws [3]). I think that any rules strict enough to be effective would also produce a device whose ability was clearly inferior to the unfettered versions (and so human competition would favor the development of the those more dangerous models). Still, the Asimov dream is a wonderful one: Imagine a willing slave, who has 1000 times your capabilities in every way. Imagine a creature who could satisfy your every safe wish (whatever that means) and still have 99.9% of its time free for other activities. There would be a new universe we never really understood, but filled with benevolent gods (though one of _my_ wishes might be to become one of them).

If the Singularity can not be prevented or confined, just how bad could the Post-Human era be? Well ... pretty bad. The physical extinction of the human race is one possibility. (Or as Eric Drexler put it of nanotechnology: Given all that such technology can do, perhaps governments would simply decide that they no longer need citizens!). Yet physical extinction may not be the scariest possibility. Again, analogies: Think of the different ways we relate to animals. Some of the crude physical abuses are implausible, yet.... In a Post-Human world there would still be plenty of niches where human equivalent automation would be desirable: embedded systems in autonomous devices, self-aware daemons in the lower functioning of larger sentients. (A strongly superhuman intelligence would likely be a Society of Mind [16] with some very competent components.) Some of these human equivalents might be used for nothing more than digital signal processing. They would be more like whales than humans. Others might be very human-like, yet with a one-sidedness, a _dedication_ that would put them in a mental hospital in our era. Though none of these creatures might be flesh-and-blood humans, they might be the closest things in the new enviroment to what we call human now. (I. J. Good had something to say about this, though at this late date the advice may be moot: Good [12] proposed a "Meta-Golden Rule", which might be paraphrased as "Treat your inferiors as you would be treated by your superiors." It's a wonderful, paradoxical idea (and most of my friends don't believe it) since the game-theoretic payoff is so hard to articulate. Yet if we were able to follow it, in some sense that might say something about the plausibility of such kindness in this universe.)

I have argued above that we cannot prevent the Singularity, that its coming is an inevitable consequence of the humans' natural competitiveness and the possibilities inherent in technology. And yet ... we are the initiators. Even the largest avalanche is triggered by small things. We have the freedom to establish initial conditions, make things happen in ways that are less inimical than others. Of course (as with starting avalanches), it may not be clear what the right guiding nudge really is:

Other Paths to the Singularity: Intelligence Amplification_

When people speak of creating superhumanly intelligent beings, they are usually imagining an AI project. But as I noted at the beginning of this paper, there are other paths to superhumanity. Computer networks and human-computer interfaces seem more mundane than AI, and yet they could lead to the Singularity. I call this contrasting approach Intelligence Amplification (IA). IA is something that is proceeding very naturally, in most cases not even recognized by its developers for what it is. But every time our ability to access information and to communicate it to others is improved, in some sense we have achieved an increase over natural intelligence. Even now, the team of a PhD human and good computer workstation (even an off-net workstation!) could probably max any written intelligence test in existence.

And it's very likely that IA is a much easier road to the achievement of superhumanity than pure AI. In humans, the hardest development problems have already been solved. Building up from within ourselves ought to be easier than figuring out first what we really are and then building machines that are all of that. And there is at least conjectural precedent for this approach. Cairns-Smith [6] has speculated that biological life may have begun as an adjunct to still more primitive life based on crystalline growth. Lynn Margulis (in [15] and elsewhere) has made strong arguments that mutualism is a great driving force in evolution.

Note that I am not proposing that AI research be ignored or less funded. What goes on with AI will often have applications in IA, and vice versa. I am suggesting that we recognize that in network and interface research there is something as profound (and potential wild) as Artificial Intelligence. With that insight, we may see projects that are not as directly applicable as conventional interface and network design work, but which serve to advance us toward the Singularity along the IA path.

Here are some possible projects that take on special significance, given the IA point of view:

Human/computer team automation: Take problems that are normally considered for purely machine solution (like hill-climbing problems), and design programs and interfaces that take a advantage of humans' intuition and available computer hardware. Considering all the bizarreness of higher dimensional hill-climbing problems (and the neat algorithms that have been devised for their solution), there could be some very interesting displays and control tools provided to the human team member.

Develop human/computer symbiosis in art: Combine the graphic generation capability of modern machines and the esthetic sensibility of humans. Of course, there has been an enormous amount of research in designing computer aids for artists, as labor saving tools. I'm suggesting that we explicitly aim for a greater merging of competence, that we explicitly recognize the cooperative approach that is possible. Karl Sims [23] has done wonderful work in this direction.

Allow human/computer teams at chess tournaments. We already have programs that can play better than almost all humans. But how much work has been done on how this power could be used by a human, to get something even better? If such teams were allowed in at least some chess tournaments, it could have the positive effect on IA research that allowing computers in tournaments had for the corresponding niche in AI.

Develop interfaces that allow computer and network access without requiring the human to be tied to one spot, sitting in front of a computer. (This is an aspect of IA that fits so well with known economic advantages that lots of effort is already being spent on it.)

Develop more symmetrical decision support systems. A popular research/product area in recent years has been decision support systems. This is a form of IA, but may be too focused on systems that are oracular. As much as the program giving the user information, there must be the idea of the user giving the program guidance.

Use local area nets to make human teams that really work (ie, are more effective than their component members). This is generally the area of "groupware", already a very popular commercial pursuit. The change in viewpoint here would be to regard the group activity as a combination organism. In one sense, this suggestion might be regarded as the goal of inventing a "Rules of Order" for such combination operations. For instance, group focus might be more easily maintained than in classical meetings. Expertise of individual human members could be isolated from ego issues such that the contribution of different members is focused on the team project. And of course shared data bases could be used much more conveniently than in conventional committee operations. (Note that this suggestion is aimed at team operations rather than political meetings. In a political setting, the automation described above would simply enforce the power of the persons making the rules!)

Exploit the worldwide Internet as a combination human/machine tool. Of all the items on the list, progress in this is proceeding the fastest and may run us into the Singularity before anything else. The power and influence of even the present-day Internet is vastly underestimated. For instance, I think our contemporary computer systems would break under the weight of their own complexity if it weren't for the edge that the USENET "group mind" gives the system administration and support people! The very anarchy of the worldwide net development is evidence of its potential. As connectivity and bandwidth and archive size and computer speed all increase, we are seeing something like Lynn Margulis' [15] vision of the biosphere as data processor recapitulated, but at a million times greater speed and with millions of humanly intelligent agents (ourselves).

The above examples illustrate research that can be done within the context of contemporary computer science departments. There are other paradigms. For example, much of the work in Artificial Intelligence and neural nets would benefit from a closer connection with biological life. Instead of simply trying to model and understand biological life with computers, research could be directed toward the creation of composite systems that rely on biological life for guidance or for the providing features we don't understand well enough yet to implement in hardware. A long-time dream of science-fiction has been direct brain to computer interfaces [2] [29]. In fact, there is concrete work that can be done (and is being done) in this area:

Limb prosthetics is a topic of direct commercial applicability. Nerve to silicon transducers can be made [14]. This is an exciting, near-term step toward direct communication.

Direct links into brains seem feasible, if the bit rate is low: given human learning flexibility, the actual brain neuron targets might not have to be precisely selected. Even 100 bits per second would be of great use to stroke victims who would otherwise be confined to menu-driven interfaces.

Plugging in to the optic trunk has the potential for bandwidths of 1 Mbit/second or so. But for this, we need to know the fine-scale architecture of vision, and we need to place an enormous web of electrodes with exquisite precision. If we want our high bandwidth connection to be _in addition_ to what paths are already present in the brain, the problem becomes vastly more intractable. Just sticking a grid of high-bandwidth receivers into a brain certainly won't do it. But suppose that the high-bandwidth grid were present while the brain structure was actually setting up, as the embryo develops. That suggests:

Animal embryo experiments. I wouldn't expect any IA success in the first years of such research, but giving developing brains access to complex simulated neural structures might be very interesting to the people who study how the embryonic brain develops. In the long run, such experiments might produce animals with additional sense paths and interesting intellectual abilities.

Originally, I had hoped that this discussion of IA would yield some clearly safer approaches to the Singularity. (After all, IA allows our participation in a kind of transcendance.) Alas, looking back over these IA proposals, about all I am sure of is that they should be considered, that they may give us more options. But as for safety ... well, some of the suggestions are a little scarey on their face. One of my informal reviewers pointed out that IA for individual humans creates a rather sinister elite. We humans have millions of years of evolutionary baggage that makes us regard competition in a deadly light. Much of that deadliness may not be necessary in today's world, one where losers take on the winners' tricks and are co-opted into the winners' enterprises. A creature that was built _de novo_ might possibly be a much more benign entity than one with a kernel based on fang and talon. And even the egalitarian view of an Internet that wakes up along with all mankind can be viewed as a nightmare [26].

The problem is not simply that the Singularity represents the passing of humankind from center stage, but that it contradicts our most deeply held notions of being. I think a closer look at the notion of strong superhumanity can show why that is.

Strong Superhumanity and the Best We Can Ask for

Suppose we could tailor the Singularity. Suppose we could attain our most extravagant hopes. What then would we ask for: That humans themselves would become their own successors, that whatever injustice occurs would be tempered by our knowledge of our roots. For those who remained unaltered, the goal would be benign treatment (perhaps even giving the stay-behinds the appearance of being masters of godlike slaves). It could be a golden age that also involved progress (overleaping Stent's barrier). Immortality (or at least a lifetime as long as we can make the universe survive [10] [4]) would be achievable.

But in this brightest and kindest world, the philosophical problems themselves become intimidating. A mind that stays at the same capacity cannot live forever; after a few thousand years it would look more like a repeating tape loop than a person. (The most chilling picture I have seen of this is in [18].) To live indefinitely long, the mind itself must grow ... and when it becomes great enough, and looks back ... what fellow-feeling can it have with the soul that it was originally? Certainly the later being would be everything the original was, but so much vastly more. And so even for the individual, the Cairns-Smith or Lynn Margulis notion of new life growing incrementally out of the old must still be valid.

This "problem" about immortality comes up in much more direct ways. The notion of ego and self-awareness has been the bedrock of the hardheaded rationalism of the last few centuries. Yet now the notion of self-awareness is under attack from the Artificial Intelligence people ("self-awareness and other delusions"). Intelligence Amplification undercuts our concept of ego from another direction. The post-Singularity world will involve extremely high-bandwidth networking. A central feature of strongly superhuman entities will likely be their ability to communicate at variable bandwidths, including ones far higher than speech or written messages. What happens when pieces of ego can be copied and merged, when the size of a selfawareness can grow or shrink to fit the nature of the problems under consideration? These are essential features of strong superhumanity and the Singularity. Thinking about them, one begins to feel how essentially strange and different the Post-Human era will be -- _no matter how cleverly and benignly it is brought to be_.

From one angle, the vision fits many of our happiest dreams: a time unending, where we can truly know one another and understand the deepest mysteries. From another angle, it's a lot like the worst- case scenario I imagined earlier in this paper.

Which is the valid viewpoint? In fact, I think the new era is simply too different to fit into the classical frame of good and evil. That frame is based on the idea of isolated, immutable minds connected by tenuous, low-bandwith links. But the post-Singularity world _does_ fit with the larger tradition of change and cooperation that started long ago (perhaps even before the rise of biological life). I think there _are_ notions of ethics that would apply in such an era. Research into IA and high-bandwidth communications should improve this understanding. I see just the glimmerings of this now [32]. There is Good's Meta-Golden Rule; perhaps there are rules for distinguishing self from others on the basis of bandwidth of connection. And while mind and self will be vastly more labile than in the past, much of what we value (knowledge, memory, thought) need never be lost. I think Freeman Dyson has it right when he says [9]: "God is what mind becomes when it has passed beyond the scale of our comprehension."

[I wish to thank John Carroll of San Diego State University and Howard Davidson of Sun Microsystems for discussing the draft version of this paper with me.]

Annotated Sources [and an occasional plea for bibliographical help]

[1] Alfve'n, Hannes, writing as Olof Johanneson, _The End of Man?_, Award Books, 1969 earlier published as "The Tale of the Big Computer", Coward-McCann, translated from a book copyright 1966 Albert Bonniers Forlag AB with English translation copyright 1966 by Victor Gollanz, Ltd.

[2] Anderson, Poul, "Kings Who Die", _If_, March 1962, p8-36. Reprinted in _Seven Conquests_, Poul Anderson, MacMillan Co., 1969.

[3] Asimov, Isaac, "Runaround", _Astounding Science Fiction_, March 1942, p94. Reprinted in _Robot Visions_, Isaac Asimov, ROC, 1990. Asimov describes the development of his robotics stories in this book.

[4] Barrow, John D. and Frank J. Tipler, _The Anthropic Cosmological Principle_, Oxford University Press, 1986.

[5] Bear, Greg, "Blood Music", _Analog Science Fiction-Science Fact_, June, 1983. Expanded into the novel _Blood Music_, Morrow, 1985.

[6] Cairns-Smith, A. G., _Seven Clues to the Origin of Life_, Cambridge University Press, 1985.

[7] Conrad, Michael _et al._, "Towards an Artificial Brain", _BioSystems_, vol 23, pp175-218, 1989.

[8] Drexler, K. Eric, _Engines of Creation_, Anchor Press/Doubleday, 1986.

[9] Dyson, Freeman, _Infinite in All Directions_, Harper && Row, 1988.

[10] Dyson, Freeman, "Physics and Biology in an Open Universe", _Review of Modern Physics_, vol 51, pp447-460, 1979.

[11] Good, I. J., "Speculations Concerning the First Ultraintelligent Machine", in _Advances in Computers_, vol 6, Franz L. Alt and Morris Rubinoff, eds, pp31-88, 1965, Academic Press.

[12] Good, I. J., [Help! I can't find the source of Good's Meta-Golden Rule, though I have the clear recollection of hearing about it sometime in the 1960s. Through the help of the net, I have found pointers to a number of related items. G. Harry Stine and Andrew Haley have written about metalaw as it might relate to extraterrestrials: G. Harry Stine, "How to Get along with Extraterrestrials ... or Your Neighbor", _Analog Science Fact- Science Fiction_, February, 1980, p39-47.] [13] Herbert, Frank, _Dune_, Berkley Books, 1985. However, this novel was serialized in _Analog Science Fiction-Science Fact_ in the 1960s.

[14] Kovacs, G. T. A. _et al._, "Regeneration Microelectrode Array for Peripheral Nerve Recording and Stimulation", _IEEE Transactions on Biomedical Engineering_, v 39, n 9, pp 893-902.

[15] Margulis, Lynn and Dorion Sagan, _Microcosmos, Four Billion Years of Evolution from Our Microbial Ancestors_, Summit Books, 1986.

[16] Minsky, Marvin, _Society of Mind_, Simon and Schuster, 1985.

[17] Moravec, Hans, _Mind Children_, Harvard University Press, 1988.

[18] Niven, Larry, "The Ethics of Madness", _If_, April 1967, pp82-108. Reprinted in _Neutron Star_, Larry Niven, Ballantine Books, 1968.

[19] Penrose, Roger, _The Emperor's New Mind_, Oxford University Press, 1989.

[20] Platt, Charles, Private Communication.

[21] Rasmussen, S. _et al._, "Computational Connectionism within Neurons: a Model of Cytoskeletal Automata Subserving Neural Networks", in _Emergent Computation_, Stephanie Forrest, ed., pp428-449, MIT Press, 1991.

[22] Searle, John R., "Minds, Brains, and Programs", in _The Behavioral and Brain Sciences_, vol 3, Cambridge University Press, 1980. The essay is reprinted in _The Mind's I_, edited by Douglas R. Hofstadter and Daniel C. Dennett, Basic Books, 1981 (my source for this reference). This reprinting contains an excellent critique of the Searle essay.

[23] Sims, Karl, "Interactive Evolution of Dynamical Systems", Thinking Machines Corporation, Technical Report Series (published in _Toward a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life_, Paris, MIT Press, December 1991.

[24] Stapledon, Olaf, _The Starmaker_, Berkley Books, 1961 (but from the date on forward, probably written before 1937).

[25] Stent, Gunther S., _The Coming of the Golden Age: A View of the End of Progress_, The Natural History Press, 1969.

[26] Swanwick Michael, _Vacuum Flowers_, serialized in _Isaac Asimov's Science Fiction Magazine_, December(?) 1986 - February 1987. Republished by Ace Books, 1988.

[27] Thearling, Kurt, "How We Will Build a Machine that Thinks", a workshop at Thinking Machines Corporation, August 24-26, 1992. Personal Communication.

[28] Ulam, S., Tribute to John von Neumann, _Bulletin of the American Mathematical Society_, vol 64, nr 3, part 2, May 1958, pp1-49.

[29] Vinge, Vernor, "Bookworm, Run!", _Analog_, March 1966, pp8-40. Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen Books, 1987.

[30] Vinge, Vernor, "True Names", _Binary Star Number 5_, Dell, 1981. Reprinted in _True Names and Other Dangers_, Vernor Vinge, Baen Books, 1987.

[31] Vinge, Vernor, First Word, _Omni_, January 1983, p10.

[32] Vinge, Vernor, To Appear [ :-) ].++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

A PROPOSAL FOR THE DARTMOUTH SUMMER RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE

J. McCarthy, Dartmouth College
M. L. Minsky, Harvard University
N. Rochester, I.B.M. Corporation
C.E. Shannon, Bell Telephone Laboratories

August 31, 1955

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

The following are some aspects of the artificial intelligence problem:

1 Automatic Computers

If a machine can do a job, then an automatic calculator can be programmed to simulate the machine. The speeds and memory capacities of present computers may be insufficient to simulate many of the higher functions of the human brain, but the major obstacle is not lack of machine capacity, but our inability to write programs taking full advantage of what we have.

2. How Can a Computer be Programmed to Use a Language

It may be speculated that a large part of human thought consists of manipulating words according to rules of reasoning and rules of conjecture. From this point of view, forming a generalization consists of admitting a new word and some rules whereby sentences containing it imply and are implied by others. This idea has never been very precisely formulated nor have examples been worked out.

3. Neuron Nets

How can a set of (hypothetical) neurons be arranged so as to form concepts. Considerable theoretical and experimental work has been done on this problem by Uttley, Rashevsky and his group, Farley and Clark, Pitts and McCulloch, Minsky, Rochester and Holland, and others. Partial results have been obtained but the problem needs more theoretical work.

4. Theory of the Size of a Calculation

If we are given a well-defined problem (one for which it is possible to test mechanically whether or not a proposed answer is a valid answer) one way of solving it is to try all possible answers in order. This method is inefficient, and to exclude it one must have some criterion for efficiency of calculation. Some consideration will show that to get a measure of the efficiency of a calculation it is necessary to have on hand a method of measuring the complexity of calculating devices which in turn can be done if one has a theory of the complexity of functions. Some partial results on this problem have been obtained by Shannon, and also by McCarthy.

5. Self-lmprovement

Probably a truly intelligent machine will carry out activities which may best be described as self-improvement. Some schemes for doing this have been proposed and are worth further study. It seems likely that this question can be studied abstractly as well.

6. Abstractions

A number of types of ``abstraction'' can be distinctly defined and several others less distinctly. A direct attempt to classify these and to describe machine methods of forming abstractions from sensory and other data would seem worthwhile.

7. Randomness and Creativity

A fairly attractive and yet clearly incomplete conjecture is that the difference between creative thinking and unimaginative competent thinking lies in the injection of a some randomness. The randomness must be guided by intuition to be efficient. In other words, the educated guess or the hunch include controlled randomness in otherwise orderly thinking.

In addition to the above collectively formulated problems for study, we have asked the individuals taking part to describe what they will work on. Statements by the four originators of the project are attached.

We propose to organize the work of the group as follows.

Potential participants will be sent copies of this proposal and asked if they would like to work on the artificial intelligence problem in the group and if so what they would like to work on. The invitations will be made by the organizing committee on the basis of its estimate of the individual's potential contribution to the work of the group. The members will circulate their previous work and their ideas for the problems to be attacked during the months preceding the working period of the group.

During the meeting there will be regular research seminars and opportunity for the members to work individually and in informal small groups.

The originators of this proposal are:

1. C. E. Shannon, Mathematician, Bell Telephone Laboratories. Shannon developed the statistical theory of information, the application of propositional calculus to switching circuits, and has results on the efficient synthesis of switching circuits, the design of machines that learn, cryptography, and the theory of Turing machines. He and J. McCarthy are co-editing an Annals of Mathematics Study on ``The Theory of Automata'' .

2. M. L. Minsky, Harvard Junior Fellow in Mathematics and Neurology. Minsky has built a machine for simulating learning by nerve nets and has written a Princeton PhD thesis in mathematics entitled, ``Neural Nets and the Brain Model Problem'' which includes results in learning theory and the theory of random neural nets.

3. N. Rochester, Manager of Information Research, IBM Corporation, Poughkeepsie, New York. Rochester was concerned with the development of radar for seven years and computing machinery for seven years. He and another engineer were jointly responsible for the design of the IBM Type 701 which is a large scale automatic computer in wide use today. He worked out some of the automatic programming techniques which are in wide use today and has been concerned with problems of how to get machines to do tasks which previously could be done only by people. He has also worked on simulation of nerve nets with particular emphasis on using computers to test theories in neurophysiology.

4. J. McCarthy, Assistant Professor of Mathematics, Dartmouth College. McCarthy has worked on a number of questions connected with the mathematical nature of the thought process including the theory of Turing machines, the speed of computers, the relation of a brain model to its environment, and the use of languages by machines. Some results of this work are included in the forthcoming ``Annals Study'' edited by Shannon and McCarthy. McCarthy's other work has been in the field of differential equations.

The Rockefeller Foundation is being asked to provide financial support for the project on the following basis:

1. Salaries of $1200 for each faculty level participant who is not being supported by his own organization. It is expected, for example, that the participants from Bell Laboratories and IBM Corporation will be supported by these organizations while those from Dartmouth and Harvard will require foundation support.

2. Salaries of $700 for up to two graduate students.

3. Railway fare for participants coming from a distance.

4. Rent for people who are simultaneously renting elsewhere.

5. Secretarial expenses of $650, $500 for a secretary and $150 for duplicating expenses.

6. Organization expenses of $200. (Includes expense of reproducing preliminary work by participants and travel necessary for organization purposes.

7. Expenses for two or three people visiting for a short time.

#& # Estimated Expenses 6 salaries of 1200 & $7200 2 salaries of 700 & 1400 8 traveling and rent expenses averaging 300 & 2400 Secretarial and organizational expense & 850 Additional traveling expenses & 600 Contingencies & 550 &---- & $13,500

I would like to devote my research to one or both of the topics listed below. While I hope to do so, it is possible that because of personal considerations I may not be able to attend for the entire two months. I, nevertheless, intend to be there for whatever time is possible.

1. Application of information theory concepts to computing machines and brain models. A basic problem in information theory is that of transmitting information reliably over a noisy channel. An analogous problem in computing machines is that of reliable computing using unreliable elements. This problem has been studies by von Neumann for Sheffer stroke elements and by Shannon and Moore for relays; but there are still many open questions. The problem for several elements, the development of concepts similar to channel capacity, the sharper analysis of upper and lower bounds on the required redundancy, etc. are among the important issues. Another question deals with the theory of information networks where information flows in many closed loops (as contrasted with the simple one-way channel usually considered in communication theory). Questions of delay become very important in the closed loop case, and a whole new approach seems necessary. This would probably involve concepts such as partial entropies when a part of the past history of a message ensemble is known.

2. The matched environment - brain model approach to automata. In general a machine or animal can only adapt to or operate in a limited class of environments. Even the complex human brain first adapts to the simpler aspects of its environment, and gradually builds up to the more complex features. I propose to study the synthesis of brain models by the parallel development of a series of matched (theoretical) environments and corresponding brain models which adapt to them. The emphasis here is on clarifying the environmental model, and representing it as a mathematical structure. Often in discussing mechanized intelligence, we think of machines performing the most advanced human thought activities-proving theorems, writing music, or playing chess. I am proposing here to start at the simple and when the environment is neither hostile (merely indifferent) nor complex, and to work up through a series of easy stages in the direction of these advanced activities.

It is not difficult to design a machine which exhibits the following type of learning. The machine is provided with input and output channels and an internal means of providing varied output responses to inputs in such a way that the machine may be ``trained'' by a ``trial and error'' process to acquire one of a range of input-output functions. Such a machine, when placed in an appropriate environment and given a criterior of ``success'' or ``failure'' can be trained to exhibit ``goal-seeking'' behavior. Unless the machine is provided with, or is able to develop, a way of abstracting sensory material, it can progress through a complicated environment only through painfully slow steps, and in general will not reach a high level of behavior.

Now let the criterion of success be not merely the appearance of a desired activity pattern at the output channel of the machine, but rather the performance of a given manipulation in a given environment. Then in certain ways the motor situation appears to be a dual of the sensory situation, and progress can be reasonably fast only if the machine is equally capable of assembling an ensemble of ``motor abstractions'' relating its output activity to changes in the environment. Such ``motor abstractions'' can be valuable only if they relate to changes in the environment which can be detected by the machine as changes in the sensory situation, i.e., if they are related, through the structure of the environrnent, to the sensory abstractions that the machine is using.

I have been studying such systems for some time and feel that if a machine can be designed in which the sensory and motor abstractions, as they are formed, can be made to satisfy certain relations, a high order of behavior may result. These relations involve pairing, motor abstractions with sensory abstractions in such a way as to produce new sensory situations representing the changes in the environment that might be expected if the corresponding motor act actually took place.

The important result that would be looked for would be that the machine would tend to build up within itself an abstract model of the environment in which it is placed. If it were given a problem, it could first explore solutions within the internal abstract model of the environment and then attempt external experiments. Because of this preliminary internal study, these external experiments would appear to be rather clever, and the behavior would have to be regarded as rather ``imaginative''

A very tentative proposal of how this might be done is described in my dissertation and I intend to do further work in this direction. I hope that by summer 1956 I wi11 have a model of such a machine fairly close to the stage of programming in a computer.

Originality in Machine Performance

In writing a program for an automatic calculator, one ordinarily provides the machine with a set of rules to cover each contingency which may arise and confront the machine. One expects the machine to follow this set of rules slavishly and to exhibit no originality or common sense. Furthermore one is annoyed only at himself when the machine gets confused because the rules he has provided for the machine are slightly contradictory. Finally, in writing programs for machines, one sometimes must go at problems in a very laborious manner whereas, if the machine had just a little intuition or could make reasonable guesses, the solution of the problem could be quite direct. This paper describes a conjecture as to how to make a machine behave in a somewhat more sophisticated manner in the general area suggested above. The paper discusses a problem on which I have been working sporadically for about five years and which I wish to pursue further in the Artificial Intelligence Project next summer.

The Process of Invention or Discovery

Living in the environment of our culture provides us with procedures for solving many problems. Just how these procedures work is not yet clear but I shall discuss this aspect of the problem in terms of a model suggested by Craik . He suggests that mental action consists basically of constructing little engines inside the brain which can simulate and thus predict abstractions relating to environment. Thus the solution of a problem which one already understands is done as follows:

The environment provides data from which certain abstractions are formed.
The abstractions together with certain internal habits or drives provide:
1. A definition of a problem in terms of desired condition to be achieved in the future, a goal.
2. A suggested action to solve the problem.
3. Stimulation to arouse in the brain the engine which corresponds to this situation.
Then the engine operates to predict what this environmental situation and the proposed reaction will lead to.
If the prediction corresponds to the goal the individual proceeds to act as indicated.

The prediction will correspond to the goal if living in the environment of his culture has provided the individual with the solution to the problem. Regarding the individual as a stored program calculator, the program contains rules to cover this particular contingency.

For a more complex situation the rules might be more complicated. The rules might call for testing each of a set of possible actions to determine which provided the solution. A still more complex set of rules might provide for uncertainty about the environment, as for example in playing tic tac toe one must not only consider his next move but the various possible moves of the environment (his opponent).

Now consider a problem for which no individual in the culture has a solution and which has resisted efforts at solution. This might be a typical current unsolved scientific problem. The individual might try to solve it and find that every reasonable action led to failure. In other words the stored program contains rules for the solution of this problem but the rules are slightly wrong.

In order to solve this problem the individual will have to do something which is unreasonable or unexpected as judged by the heritage of wisdom accumulated by the culture. He could get such behavior by trying different things at random but such an approach would usually be too inefficient. There are usually too many possible courses of action of which only a tiny fraction are acceptable. The individual needs a hunch, something unexpected but not altogether reasonable. Some problems, often those which are fairly new and have not resisted much effort, need just a little randomness. Others, often those which have long resisted solution, need a really bizarre deviation from traditional methods. A problem whose solution requires originality could yield to a method of solution which involved randomness.

In terms of Craik's S model, the engine which should simulate the environment at first fails to simulate correctly. Therefore, it is necessary to try various modifications of the engine until one is found that makes it do what is needed.

Instead of describing the problem in terms of an individual in his culture it could have been described in terms of the learning of an immature individual. When the individual is presented with a problem outside the scope of his experience he must surmount it in a similar manner.

So far the nearest practical approach using this method in machine solution of problems is an extension of the Monte Carlo method. In the usual problem which is appropriate for Monte Carlo there is a situation which is grossly misunderstood and which has too many possible factors and one is unable to decide which factors to ignore in working out analytical solution. So the mathematician has the machine making a few thousand random experiments. The results of these experiments provide a rough guess as to what the answer may be. The extension of the Monte Carlo Method is to use these results as a guide to determine what to neglect in order to simplify the problem enough to obtain an approximate analytical solution.

It might be asked why the method should include randomness. Why shouldn't the method be to try each possibility in the order of the probability that the present state of knowledge would predict for its success? For the scientist surrounded by the environment provided by his culture, it may be that one scientist alone would be unlikely to solve the problem in his life so the efforts of many are needed. If they use randomness they could all work at once on it without complete duplication of effort. If they used system they would require impossibly detailed communication. For the individual maturing in competition with other individuals the requirements of mixed strategy (using game theory terminology) favor randomness. For the machine, randomness will probably be needed to overcome the shortsightedness and prejudices of the programmer. While the necessity for randomness has clearly not been proven, there is much evidence in its favor.

The Machine With Randomness

In order to write a program to make an automatic calculator use originality it will not do to introduce randomness without using forsight. If, for example, one wrote a program so that once in every 10,000 steps the calculator generated a random number and executed it as an instruction the result would probably be chaos. Then after a certain amount of chaos the machine would probably try something forbidden or execute a stop instruction and the experiment would be over.

Two approaches, however, appear to be reasonable. One of these is to find how the brain manages to do this sort of thing and copy it. The other is to take some class of real problems which require originality in their solution and attempt to find a way to write a program to solve them on an automatic calculator. Either of these approaches would probably eventually succeed. However, it is not clear which would be quicker nor how many years or generations it would take. Most of my effort along these lines has so far been on the former approach because I felt that it would be best to master all relevant scientific knowledge in order to work on such a hard problem, and I already was quite aware of the current state of calculators and the art of programming them.

The control mechanism of the brain is clearly very different from the control mechanism in today's calculators. One symptom of the difference is the manner of failure. A failure of a calculator characteristically produces something quite unreasonable. An error in memory or in data transmission is as likely to be in the most significant digit as in the least. An error in control can do nearly anything. It might execute the wrong instruction or operate a wrong input-output unit. On the other hand human errors in speech are apt to result in statements which almost make sense (consider someone who is almost asleep, slightly drunk, or slightly feverish). Perhaps the mechanism of the brain is such that a slight error in reasoning introduces randomness in just the right way. Perhaps the mechanism that controls serial order in behavior guides the random factor so as to improve the efficiency of imaginative processes over pure randomness.

Some work has been done on simulating neuron nets on our automatic calculator. One purpose was to see if it would be thereby possible to introduce randomness in an appropriate fashion. It seems to have turned out that there are too many unknown links between the activity of neurons and problem solving for this approach to work quite yet. The results have cast some light on the behavior of nets and neurons, but have not yielded a way to solve problems requiring originality.

An important aspect of this work has been an effort to make the machine form and manipulate concepts, abstractions, generalizations, and names. An attempt was made to test a theory of how the brain does it. The first set of experiments occasioned a revision of certain details of the theory. The second set of experiments is now in progress. By next summer this work will be finished and a final report will have been written.

My program is to try next to write a program to solve problems which are members of some limited class of problems that require originality in their solution. It is too early to predict just what stage I will be in next summer, or just; how I will then define the immediate problem. However, the underlying problem which is described in this paper is what I intend to pursue. In a single sentence the problem is: how can I make a machine which will exhibit originality in its solution of problems?

1. K.J.W. Craik, The Nature of Explanation, Cambridge University Press, 1943 (reprinted 1952), p. 92.

2. K.S. Lashley, ``The Problem of Serial Order in Behavior'', in Cerebral Mechanism in Behavior, the Hixon Symposium, edited by L.A. Jeffress, John Wiley & Sons, New York, pp. 112-146, 1951.

3. D. O. Hebb, The Organization of Behavior, John Wiley & Sons, New York, 1949

During next year and during the Summer Research Project on Artificial Intelligence, I propose to study the relation of language to intelligence. It seems clear that the direct application of trial and error methods to the relation between sensory data and motor activity will not lead to any very complicated behavior. Rather it is necessary for the trial and error methods to be applied at a higher level of abstraction. The human mind apparently uses language as its means of handling complicated phenomena. The trial and error processes at a higher level frequently take the form of formulating conjectures and testing them. The English language has a number of properties which every formal language described so far lacks.

1. Arguments in English supplemented by informal mathematics can be concise.

2. English is universal in the sense that it can set up any other language within English and then use that language where it is appropriate.

3. The user of English can refer to himself in it and formulate statements regarding his progress in solving the problem he is working on.

4. In addition to rules of proof, English if completely formulated would have rules of conjecture .

The logical languages so far formulated have either been instruction lists to make computers carry out calculations specified in advance or else formalization of parts of mathematics. The latter have been constructed so as:

1. to be easily described in informal mathematics,

2. to allow translation of statements from informal mathematics into the language,

3. to make it easy to argue about whether proofs of (???)

No attempt has been made to make proofs in artificial languages as short as informal proofs. It therefore seems to be desirable to attempt to construct an artificial language which a computer can be programmed to use on problems requiring conjecture and self-reference. It should correspond to English in the sense that short English statements about the given subject matter should have short correspondents in the language and so should short arguments or conjectural arguments. I hope to try to formulate a language having these properties and in addition to contain the notions of physical object, event, etc., with the hope that using this language it will be possible to program a machine to learn to play games well and do other tasks .

The purpose of the list is to let those on it know who is interested in receiving documents on the problem. The people on the 1ist wlll receive copies of the report of the Dartmouth Summer Project on Artificial Intelligence. [1996 note: There was no report.]

The list consists of people who particlpated in or visited the Dartmouth Summer Research Project on Artificlal Intelligence, or who are known to be interested in the subject. It is being sent to the people on the 1ist and to a few others.

For the present purpose the artificial intelligence problem is taken to be that of making a machine behave in ways that would be called intelligent if a human were so behaving.

A revised list will be issued soon, so that anyone else interested in getting on the list or anyone who wishes to change his address on it should write to:

1996 note: Not all of these people came to the Dartmouth conference. They were people we thought might be interested in Artificial Intelligence.

The list consists of:

Adelson, Marvin
Hughes Aircraft Company
Airport Station, Los Angeles, CA

Ashby, W. R.
Barnwood House
Gloucester, England

Backus, John
IBM Corporation
590 Madison Avenue
New York, NY

Bernstein, Alex
IBM Corporation
590 Madison Avenue
New York, NY

Bigelow, J. H.
Institute for Advanced Studies
Princeton, NJ

Elias, Peter
R. L. E., MIT
Cambridge, MA

Duda, W. L.
IBM Research Laboratory
Poughkeepsie, NY

Davies, Paul M.
1317 C. 18th Street
Los Angeles, CA.

Fano, R. M.
R. L. E., MIT
Cambridge, MA

Farley, B. G.
324 Park Avenue
Arlington, MA.

Galanter, E. H.
University of Pennsylvania
Philadelphia, PA

Gelernter, Herbert
IBM Research
Poughkeepsie, NY

Glashow, Harvey A.
1102 Olivia Street
Ann Arbor, MI.

Goertzal, Herbert
330 West 11th Street
New York, New York

Hagelbarger, D.
Bell Telephone Laboratories
Murray Hill, NJ

Miller, George A.
Memorial Hall
Harvard University
Cambridge, MA.

Harmon, Leon D.
Bell Telephone Laboratories
Murray Hill, NJ

Holland, John H.
E. R. I.
University of Michigan
Ann Arbor, MI

Holt, Anatol
7358 Rural Lane
Philadelphia, PA

Kautz, William H.
Stanford Research Institute
Menlo Park, CA

Luce, R. D.
427 West 117th Street
New York, NY

MacKay, Donald
Department of Physics
University of London
London, WC2, England

McCarthy, John
Dartmouth College
Hanover, NH

McCulloch, Warren S.
R.L.E., M.I.T.
Cambridge, MA

Melzak, Z. A.
Mathematics Department
University of Michigan
Ann Arbor, MI

Minsky, M. L.
112 Newbury Street
Boston, MA

More, Trenchard
Department of Electrical Engineering
MIT
Cambridge, MA

Nash, John
Institute for Advanced Studies
Princeton, NJ

Newell, Allen
Department of Industrial Administration
Carnegie Institute of Technology
Pittsburgh, PA

Robinson, Abraham
Department of Mathematics
University of Toronto
Toronto, Ontario, Canada

Rochester, Nathaniel
Engineering Research Laboratory
IBM Corporation
Poughkeepsie, NY

Rogers, Hartley, Jr.
Department of Mathematics
MIT
Cambridge, MA.

Rosenblith, Walter
R.L.E., M.I.T.
Cambridge, MA.

Rothstein, Jerome
21 East Bergen Place
Red Bank, NJ

Sayre, David
IBM Corporation
590 Madison Avenue
New York, NY

Schorr-Kon, J.J.
C-380 Lincoln Laboratory, MIT
Lexington, MA

Shapley, L.
Rand Corporation
1700 Main Street
Santa Monica, CA

Schutzenberger, M.P.
R.L.E., M.I.T.
Cambridge, MA

Selfridge, O. G.
Lincoln Laboratory, M.I.T.
Lexington, MA

Shannon, C. E.
R.L.E., M.I.T.
Cambridge, MA

Shapiro, Norman
Rand Corporation
1700 Main Street
Santa Monica, CA

Simon, Herbert A.
Department of Industrial Administration
Carnegie Institute of Technology
Pittsburgh, PA

Solomonoff, Raymond J.
Technical Research Group
17 Union Square West
New York, NY

Steele, J. E., Capt. USAF
Area B., Box 8698
Wright-Patterson AFB
Ohio

Webster, Frederick
62 Coolidge Avenue
Cambridge, MA

Moore, E. F.
Bell Telephone Laboratory
Murray Hill, NJ

Kemeny, John G.
Dartmouth College
Hanover, NH