Here's my definition: Computer intelligence will progress and at some future time it will surpass human intelligence. When that occurs, the relationship between humans and computers will become unpredictable.
My definition relies on these assumptions:
AI progress will be faster than improvements in biology - so computers will outpace the intelligence gains of humans.
AI progress will also be faster than improvements in biological and machine integrations.
Traditional evolutionary theory views organisms as vehicles for passing forward their DNA. Natural selection will ensure the fittest DNA is propagated. Will computer programs leave humans and other organisms in the dust and relegate these life forms to becoming a mere blip on the evolutionary timeline?
If we assume an AI singularity is an invariant feature of carbon-based evolution then we can easily conclude it is not a good strategy to limit SETI investigations to habitable planets.
Most of the advances in intelligence will occur after the AI singularity by computer programs, not by humans. Computer programs have no need for the support systems such as oxygen and water and food that are required by humans as posited by the "Drake equation".
The bulk of intelligence in our universe will be located in non-human environments populated with computer programs.
When Will the AI Singularity Occur?
We could say something like this: It's anyone's guess.
However, the opinions of engineers who have tracked over decades the growth of AI, computer hardware, neuroscience, nanotechnology, and other related disciplines should be given greater credence than the estimates of casual observers. See, Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology, Viking Press 2005.
My casual observer estimate is the AI singularity will not occur for another hundred years - so I will opine 2110. A computer applet can beat me in chess 9 out of 10 games, personal smartphone assistants are improving, knowledge navigators are being built, a program can be excellent at playing the game Jeopardy, and of course algorithms for many applications, such as flying planes, managing airport flight control, and high-frequency stock-market trading are constantly improving.
Other AI achievements touted by Ray Kurzweil include:
Character recognition
Why does Google's reCAPTCHA depend on people to interpret text?
Speech recognition
Why do smartphone personal assistants understand extremely few utterances?
Machine vision
Why aren't there AI tools for people who are blind to navigate inside buildings and outdoors on streets, stores, train stations, etc?
Speech synthesis
Why are smartphone personal assistants capable of uttering extremely few sentences?
Robotics
Currently, a mindless disk can vacuum a room by traversing random patterns and bumping into furniture.
Data mining
In a rigorous experiment, testing the quality of Google and Yahoo search engine results (without displaying the bells and whistles such as Google Instant and Suggest) sitting people in front of search engines from 2000 and 2013 would users find a difference? I doubt it.
Medical informatics
No one goes to a clinic and gets a check-up by a robot.
In my opinion, AI should use the following standard as a substitute for "When will AI surpass human intelligence?"
Here's my proposed standard:
How intelligent is current AI overall in aggregate compared to a human in a given class-grade X in an average US public school?
I'm going to venture this speculative guess: AI is currently at the level of a human who is in first-grade.
A first grade student has these abilities lacking by AI:
Has knowledge about people and objects in the world and can interact intelligently with them.
Can learn and understand educational subject matters.
Can converse intelligently with teachers and friends about course work.
Can ask intelligent questions.
The prevailing expert's near-term estimate for the occurrence of the AI singularity is in my opinion logically flawed for several reasons.
Extrapolation from Moore's Law
Based on Moore's Law (the number of transistors or performance of a computer chip doubles every two years) futurologists say that many other areas of technology similarly improve at an exponential rate. In my opinion this is an error and a false generalization.
A glaring example of this incorrect reasoning can be seen in the image labeled "Growth in Supercomputer Power" on page 72 of The Singularity Is Near.
Here Ray Kurzweil says:
"Moore's Law narrowly refers to the number of transistors on an integrated circuit of fixed size and sometimes has been expressed even more narrowly in terms of transistor feature size. But the most appropriate measure to track price-performance is computational speed per unit cost, an index that takes into account many levels of "cleverness" (innovation, which is to say, technological evolution). In addition to all of the invention involved in integrated circuits, there are multiple layers of improvement in computer design (for example, pipelining, parallel processing, instruction look-ahead, instruction and memory caching, and many others)"
Computational power comes from a combination of hardware and software. In chapter four "Achieving the Software of Human Intelligence: How to Reverse Engineer the Human Brain" Ray explores the complexity of the human brain mainly from the perspective of biology. On page 128 he notes: "There are more than fifty thousand neuroscientists in the world, writing articles for more than three hundred journals."
Unlike the documented growth in hardware improvements, the trend for improvements in AI software is not established. Here are two of the issues discussed in the chapter:
Development productivity: Ray says on page 313: "I estimate the doubling time of software development productivity to be approximately six years, which is slower than the doubling time for processor price-performance, which is approximately one year today."
A doubling time of six years for software development time is surely a lot slower than the doubling graph for hardware.
Algorithms Acceleration: Ray says on page 313: "Dramatic improvements have taken place in the speed and efficiency of software algorithms (on constant hardware). .... These improvements vary depending on the problem, but are nonetheless pervasive."
Intelligent algorithms are the key to the singularity, and language such as "Dramatic improvements have taken place" does not convince me the improvements are increasing exponentially. A limit to software development that I didn't see discussed is the finite number of developers. A lot of improvements in software have occurred because of the recent inflow of people into the programming profession. The finite number of people in the world places a limit on the available number of programmers who can write software libraries and create supporting tools.