From LessWrongWiki, a seed AI
[I]mproves itself by recursively rewriting its own source code without human intervention.
Some would even say this could bring about an Intelligence Explosion. Many of these sources mention that it recursively "improves" itself. What exactly does that mean? How do you define "improve"?
Many people say that since it "improves" itself recursively, the jumps in intelligence would be ever-increasing. However, they also mention that an AI like this would have a goal; for example maximizing knowledge or maximizing human happiness. How would this translate into maximizing of the ability to maximize? You could also maximize for speed, let's say. How do you measure speed? A test, like the TSP? But then wouldn't the machine optimize specifically for solving the TSP, and nothing else? How would that translate into an increasing ability to optimize itself?
In other words, what would one maximize for in an implementation of Seed AI?