To begin, run update81 to copy the starting point files into your home directory (cs81/labs/4/).

Intelligent Adaptive Curiosity (IAC) was developed by Pierre-Yves Oudeyer, Frederic Kaplan, and Verena Hafner with the goal of providing a robot with an intrinsic motivation system that pushes it to focus on situations that maximize its learning progress. IAC contains a memory that is sub-divided into sensorimotor regions. Each region contains:

• An expert, that is trained to predict the next sensory state given the current sensorimotor state: SM(t) -> S(t+1)
• A list of exemplars for training the expert.
• A list of past prediction errors made by the expert.

In the original description, IAC's memory is organized hierarchically as a tree. Initially all exemplars are members of a single region. When the region grows too large it is split into two new regions. The split is made based on a feature of the SM(t) component of the exemplars such that the sum of the variances of the S(t+1) component, weighted by the number of exemplars in that new region, is minimized. This process continues recursively on any region that grows too large.

The main processing loop of IAC works as follows:

• Sense the current situation, S(t).
• Create a list of potential actions. If the action space is continuous, generate a sample of candidate actions.
• For each candidate action, M(t), query the IAC memory with SM(t), and determine the associated region's learning progress.
• With some high probability, choose the action whose region exhibits the maximal learning progress, otherwise choose a random action.
• Execute the chosen action on the robot.

In their paper Intrinsic motivation systems for autonomous mental development, Oudeyer, Kaplan, and Hafner describe an experiment that is designed to test the effectiveness of IAC's goal of maximizing learning progress. One part of the domain is easy to learn, one part of the domain is more complex, and another part is unlearnable. We should expect to see IAC first focus on the easy portion, then focus on the more complex portion, while ignoring the unlearnable portion.

The robot is placed in an environment with a smart toy. The robot can command it's left and right wheels and also emit a frequency that makes the toy move. When the frequency is:

• [0.00:0.33] the toy moves randomly.
• [0.34:0.66] the toy stops moving.
• [0.67:1.00] the toy jumps onto the robot.

The robot senses its distance from the toy. Therefore SM(t) consists of four items: leftWheelSpeed, rightWheelSpeed, frequency, and currentDistance and S(t+1) consists of one item: nextDistance.

The paper describes an experiment that lasts for 5000 epochs. In the first 250 time steps, the robot emits all three frequencies equally. This is when the initial region first splits. Next the robot emits both the second and third frequencies, but then another split occurs at time step 500. From then until about epoch 3000, the robot emits the third frequency (the easiest one to predict) about 90% of the time. From that point on it focuses on emitting the second frequency (the more complex one to predict) about 85% of the time. It never spends more that 10% of the time on the first frequency, which is unlearnable.

In reproducing the above experiment, a number of decisions must be made:

• How should the experts be implemented? We are using a two-layer neural network. The paper does not specify the type of expert used.
• When the action space is infinite, how should the sampling be done? Should you guarantee that every region is represented in the sample?
• What probability should be used for choosing random actions versus those with maximal learning progress? The paper states that random actions were chosen 35% of the time

To start the experiment do: python iac.py

This will generate a number of data files recording the percentage of time the robot emits each of the three frequencies during every 50 time steps. These data files are updated continuously and can be viewed while the experiment is running by doing: xgraph -ly 0,1 -P *.data