self_modeling_and_motivated_value_selection

From billh@ssec.wisc.edu Sat Feb 7 11:08:22 2015

Date: Sat, 7 Feb 2015 11:08:05 -0600 (CST)

From: Bill Hibbard <billh@ssec.wisc.edu>

To: Benja Fallenstein <benja.fallenstein@gmail.com>

Cc: "researchassociates@intelligence.org" <researchassociates@intelligence.org>

Subject: Re: Weekly report 2015-01-19 through 2015-01-25

Hi Benja,

I enjoyed seeing you, Nate and Kaj at AAAI-15.

On Tue, 27 Jan 2015, Benja Fallenstein wrote:

> Also, I got to chat again with Bill Hibbard about his

> idea for -- translating into my own way of thinking

> about it -- using a bounded resource variant of AIXI

> for some types of logical uncertainty tasks. Roughly,

> the idea is illustrated by the following example:

> Suppose that every time you upgrade the computing

> hardware you run on, your average score increases

> significantly. A probabilistic environment model

> which is far too computationally constrained to

> simulate the real world (which contains the real you)

> might still be able to make probabilistic predictions

> about your score if you buy additional hardware that

> reflect this relationship. Bill had a version of this

> system on his poster, and thinking about that and

> talking to him about it helped me understand it

> somewhat better than I had before.

The self-modeling agent framework proposes

to address several problems. As you note,

one is predicting the value of increases to

agent resources.

Another is "inconsistency between the agent's

utility function and its definition." One case

of this problem is what Armstrong calls

"motivated value selection" and describes as

"a conflict between agents learning their

future values and following their current

values." In the self-modeling framework the

agent learns a function from interaction

history and proposed action to value. All

elements of the agent defintion can be included

in the function to be learned, including

evolution of the agent's environment model and

utility function as interaction history length

increases. Thus this framework can avoid any

"inconsistency between the agent's utility

function and its definition."

As defined in Section 8.4 of my book

(http://arxiv.org/abs/1411.1373), the self-

modeling framework proposes to address the

problem of being predicted by other agents in

the environment by learning when to choose

actions stochastically.

Best wishes,

Bill