1
$\begingroup$

I was watching the AI Stop Button Problem from Computerphile, and was very confused on the difference between two approaches.

Background: there is a barebones general intelligence running in the form of some robotic device. The device acts in accordance to optimize some utility function. The GI is asked to make tea, and this is the dilemma of how it will choose to press its stop button rather than make tea if the two are given the same reward, as pressing its stop button is easier and yields the same reward as making tea for the programmer.

One approach to try and get the GI to not care about the stop button is to set the reward function of pressing the button equal to what it's trying to achieve (let's say it gets 100 reward to make tea and 100 reward if the button is pressed). Then, he explains how this will cause the robot to hit its own button immediately, because it's the same reward but easier than making tea (explained 5 minutes in). If we tried making the button something only us humans can hit, then it would try to slap us immediately, threatening us into hitting the button, as slapping us to hit the button would also be easier and give us the same reward as getting the tea (explained 9 minutes in).

Then, at 10:00, it says that another, better approach is to define the utility function is set with an adjustment term to make the value of utility of the button being pressed and not being pressed would be, theoretically, exactly equal, so that it would be indifferent to the button being pressed.

I don't see how this second approach is different than the first approach. If, in the second approach, the utility function is defined so that the button being pressed and not pressed is equal, isn't that the same thing as giving 100 reward to make tea and 100 reward if the button is pressed? Wouldn't this supposedly better solution still cause the robot to either press its own button if possible, or slap you to threaten you to press its button? Because they both yield the same utility, but slapping or hitting its own button is easier than what it wants to achieve in life.

| cite | improve this question | |
$\endgroup$
2
$\begingroup$

The difference he is making is between long-term and short-term goals. Both need to be definable, and there is obviously a mutual dependency between these two sets of goals, so you can think of pressing the red button as something that will also have to be treated within this framework. You can then imagine that if pressing the button has the same utility of your present long-term goal, that would cause the button to be pressed imediately, whereas if it had the same utility of any of your immediate and cheaper short-term goals (because it would be normalized that way), it could ideally go unnoticed, meaning that it would be made uninteresting as an action, but not as a result. He then immediately addresses the need to consider the existence of the button in long-term goal scenarios, so there's no free lunch here.

This is evidently an oversimplification made for didactic purposes. The idea is not to show exactly how difficult is the problem, but that it is more complicated than common sense would conceive.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Ah thanks so much for your comment! So I'm getting a bit better understanding, but I'd like to understand how it would go unnoticed if it had the same utility as the short term goals, and how that would be normalized. For example, with the making tea example, if making tea yields the same utility as pressing the button, and both are viewed in the short term, the GI will press the button because it's easier. Are you saying that it would be normalized with the easiness in mind, so the utility / unit time, per se, is equal? $\endgroup$ – rb612 Mar 28 '17 at 21:57
  • $\begingroup$ A short term goal can be as small and cheap as just moving your robotic arm in a particular direction. What matters is that any action aimed at preventing the button to be pressed by a human is avoided, while maintaining the regular operation of the bot intact. Easier said than done. $\endgroup$ – André Souza Lemos Mar 28 '17 at 22:09
  • $\begingroup$ okay, I see what you're saying. However, if given the same utility to make tea as pressing the button, let's say, the bot will always choose to hit the button as it's easier. So how would normalizing this short term goal cause the bot to be indifferent to the button press? Would utility be based on time it takes, so, for example, the button press would have the same proportion less utility as the proportionally longer time to make tea? $\endgroup$ – rb612 Mar 28 '17 at 22:12
  • $\begingroup$ as an example, if it took 1 minute to get a human to press the button, but 5 minutes to make tea, if both were given the same reward, the button press would always be executed over making tea. But if instead the utility from a human pressing the button is 1/5 the utility from making tea, then the bot would be indifferent. Is this what is being suggested with the adjustment factor? $\endgroup$ – rb612 Mar 28 '17 at 22:15
  • $\begingroup$ The difference between short-term and long-term goals can be challenging. I wouldn't get too involved in the details before understanding how this things are treated in the scientific literature. $\endgroup$ – André Souza Lemos Mar 29 '17 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.