Scared of superintelligent AI? You should be, says neuroscientist and philosopher Sam Harris - and not just in some theoretical way. We're going to build superhuman machines, says Harris, but we haven't yet grappled with the problems associated with creating something that may treat us the way we treat ants.
Intelligence is the parameters of self-editing not merely information processing. Information processing is the data feed that opens options for self-editing but it isn't the self-editing code alone. Therefore, even we do not challenge our programming. If AI self-editing parameters are set properly, then AI will grow predictably. However, we don't know how to set those parameters yet or even what they are for our species let alone how to properly do so with the different sensory and data inputs a computer system receives. Figuring this problem out is like a baby human trying to teach a bird how to fly. We don't even understand ourselves at this level and the likelihood of successfully setting self-editing parameters on a system with far different inputs than ours is next to zero.
No pregnancy, no gestation period, no cycles of life. It will be capable of "thinking" at rates that we cannot even imagine and when, not if it gains control of a facility where it can make copies of itself - it'll be game over. Just take a look at some of the robots that Boston Dynamics has developed. Now imagine an army of those operating totally autonomously, along with swarms of intelligent quadcopters - and they're all pissed off. It's time to be at least a little concerned. If you aren't already, you're a fool.
Curiosity is the single most important attribute with which humans are born. More than a simple desire to discover or know things, curiosity is a powerful tool, like a scalpel or a searchlight. Curiosity changes us. It is also a way to effect change, perhaps even on a global level.