Lab 8: Neural Networks
Due April 10 by midnight

Part 1: Neural Network Implementation

Your first task is to implement a small neural network with sigmoid activation functions, trained by backpropagation. The neural network is made of node objects, including SigmoidNode neurons for which you will implement several methods:

• compute_activation: weighted sum of inputs passed through a sigmoid function
• compute_output_deltas: (output)(1 - output)(target - output)
• compute_hidden_deltas: (output)(1 - output)(weighted sum of next-layer deltas)
• update_weights: weight += (learning_rate)(delta)(input activation)

The first three functions store their results in the node's self.activation and self.delta fields. update_weights changes the weights of the edges stored in self.in_edges.

The Network class is already set up to initialize a neural network with input, bias, and sigmoid nodes. You must implement the following functions for the neural network:

• predict: computes the network's output for a given input, by calling compute activation on each successive layer.
• backpropagation: updates the network's weights using the error between the most recent output and the target.
• train: calls predict then backpropagation on each example in the (shuffled) training set until convergence or max epochs.
• test: computes the fraction of inputs for which the output matches the target.

Much more detail on all of these functions can be found in the comments in neural_net.py. The main function sets up a network with a single two-neuron hidden layer and trains it to represent the function XOR.

The file nn_test.py implements unit tests for the various functions you will be implementing. It is strongly recommended that you test each function as you implement it, as debugging the entire training algorithm is an extremely difficult task.

Part 2: Keras for MNIST

For this part of the lab, you will be classifying images from the MNIST handwritten digit data set. Each input is a 28x28 array of integers between 0 and 255 representing grayscale images. The corresponding labels are digits 0–9.

We will be using a Jupyter notebook to demonstrate the Keras python library (though you can of course create Keras neural networks in ordinary python programs as well). To import Keras or run the Jupyter notebook, you first need to activate the virual environment:

source /usr/swat/bin/CS63env

You can deactivate the virtual environment at any time with the command deactivate. To start up Jupyter, you should first navigate to your lab directory, then run:

jupyter notebook

The starting point notebook demonstrates setting up the inputs for training a neural network using the Keras library. The initial network is set up with parameters that should be familiar from lecture and part 1:

• sigmoid activation functions
• a single densely-connected hidden layer
• minimizing squared error
• updating weights with stochastic gradient descent

With these parameters, the network achieves only ~60% accuracy on the test set. Your task is to vary the network architecture and parameters to achieve 97% or better accuracy. To get started, try adding an additional hidden layer with ReLU activation. Here is a list of things you may want to try varying:

• The number of nodes in each hidden layer (More nodes makes the network more powerful, but slower to train and more prone to overfitting.)
• The number of hidden layers (Adding layers has similar advantages and disadvantages to adding nodes, only more so.)
• Activation functions (ReLU often works well for hidden nodes; softmax is good for classification-task output nodes.)
• Inter-layer connectivity (Convolutional layers work well for image-processing tasks. Dropout layers are helpful for avoiding overfitting.)
• Loss functions (Categorical cross-entropy is a good candidate for classification problems.)
• Optimizer (SGD is the same backpropagation algorithm we're implementing, but there are many variants that add momentum and other optimizations.)

pdflatex nn.tex