Description

Labs 5-6: Introduction to Machine Learning and cuDNN
Written by: Aadyot Bhatnagar

Submission:
——————————————————————————–

Instead of emailing us the solution, put a zip file in your home directory on Titan, in the format:

lab5_2019_submission.zip

and

lab6_2019_submission.zip

Your submission for each lab should be a single archive file (.zip) with your README file and all code.

Resources:
——————————————————————————–
You will be using the MNIST dataset of handwritten digits. This can be found on Titan in the directory /srv/cs179_mnist, though it is included as a default option for you. If you are running this on your own machine, you can download the dataset yourself from http://yann.lecun.com/exdb/mnist/ and just put all 4 of the files into a single directory, which you should specify using the –dir command line argument when running the executable.

Usage:
——————————————————————————–
The executable for lab 5 will be bin/dense-neuralnet. For lab 6, it will be bin/conv-neuralnet. To call them, you should use the command bin/<type>-neuralnet [–dir <mnist_directory>] [–act <“relu” or “tanh”>] The –dir argument sets the directory from which the MNIST data will be read, while the –act argument sets the activation that this neural network will use for intermediate layers (“tanh” or “relu”). The default activation is relu.

Overview:
——————————————————————————–
In labs 5 and 6, you will be writing a number of cuBLAS and cuDNN calls (as well as a couple of kernels) to fill out the missing parts of this neural network library. These calls will implement the core functionality of the forward pass and backward pass for the network. Once implemented, you will be able to run the main file to fit a neural network on the MNIST data set of handwritten digits, and evaluate your network’s out-of-sample performance on a separate subset of the data.

There are a number of TODO’s listed in the files layers.cpp and utils.cu. Your job is to fill all of these out with the necessary code (either from the CUDA libraries we discussed in class or via writing your own kernels). Some of these TODO’s will be labeled as lab 5, and others will be labeled as lab 6, indicating which assignment they are meant to be filled out for. The final executables for labs 5 and 6 will be bin/dense-neuralnet and bin/conv-neuralnet, respectively. Both executables should compile regardless of how much of lab 5 or 6 has been completed, but obviously they won’t run correctly until the relevant code has been written completely.

Unfortunately, we haven’t had the time to come up with any systematic way to help you debug your code for these assignments. That said, here are a few guidelines to follow that will make your life much easier:

1) Wrap all of your CUDA calls and CUDA library calls with the appropriate error checking macros! These are defined in helper_cuda.h and have been adapted from the CUDA examples.

2) If your code is working 100% correctly, you should expect to see the loss decreasing and accuracy increasing with every iteration of training. If your loss is increasing or turning into NaN’s, check the following:

a) Are you adding the gradient into the current values of the weights and biases instead of subtracting them (don’t do gradient ascent instead of descent, since we want to MINIMIZE the loss)?

b) Make sure you’re not adding the gradients with respect to the data (i.e. the input and output minibatches of a given layer) into the
data itself!

c) Especially for lab 5, make sure that your cuBLAS calls match the linear algebra operations described in the slides. We’ve made an effort to have notation be relatively consistent between the lecture slides and
the code.

3) Refer to the lecture slides frequently! There are slides in each of lectures 15 and 17 (for labs 5 and 6 respectively) that outline the actual algorithms you need to implement.