Rust Neural Network Implementation

Introduction

This project is a Rust implementation of a neural network built from scratch, without using machine learning frameworks such as PyTorch or TensorFlow.

The implementation is inspired by the video Building a neural network from scratch. It was developed as a solution for the Digit Recognizer competition.

The goal of the competition is to correctly classify handwritten digits from the MNIST dataset, which contains tens of thousands of labeled handwritten digit images.

The video and its reference notebook were used as references for the implementation. The notebook was also used to generate expected values for the automated tests.

Multi Layer Perceptron

Multi Layer Perceptron (MLP) is a type of neural network composed of multiple layers of neurons. There are at least 3 layers: one input layer (input values), one output layer (output values), and one or many hidden layers. The output of a perceptron y is a combination of all the outputs of the previous layer x_{i}, each output multiplied by a weigh w_{i}.

y = \Sigma_{i}(x_{i}w_{i} + b)

Each perceptron output is passed through an activation function before being used by the perceptron of the next layer. It introduces non-linearity into the model, allowing the network to learn and represent complex patterns in the data. The activation functions used are the Rectified Linear Unit (ReLU) for input and hidden layers and Softmax for the output layer.

ReLU(x) = \begin{cases} x &\text{if } x > 0 \ 0 &\text{else } \end{cases}

def ReLU(X):
    return np.maximum(X,0)

softmax(x_{i}) = \frac{e^{x_{i}}}{\sum_{j=1}^K e^{x_{j}}} \\ 

def softmax(X):
    exp = np.exp(X - np.max(X))
    return exp / exp.sum(axis=0)

Training

Multi-Layer Perceptron training iteratively fine-tunes the model. An iteration is composed of :

1. The forward propagation: during this step, the input data is fed through the network. The output of the last layer gives us the model prediction. We can compare the difference between the predicted output and the actual value.
2. The backward propagation: starting from the output layer, the propagates this error through the network. The goal is to evaluate, for each weight, how much this weight contributed to the final error and how we can adjust his weight to improve the final prediction. This value is called the gradient.
3. Update each weight with his computed gradient (weighted by the learning rate).

Rust implementation

Architecture

    ─ src
        ├─ model            // Multilayer perceptron struct
        ├─ train            // Training functions (forward, backward...)  
        ├─ dataset          // Dataset struct
        ├─ arguments        // Command-line argument parsing and management
        ├─ factory          // Model / dataset factories (csv / npy parser )
        └─ main.rs
    ─ tests
        ├─ python_scripts   // Scripts used to generate expected values
        └─ tests 

Crates

Rust libraries are distributed as crates. The following is a non-exhaustive list of the crates used by this project:

• nalgebra : data structure such as matrix and vector, and function to manipulate this structures
• clap : to parse arguments (i.e iteration, output destination...)
• egui and eframe : for demo gui
• csv : to parse csv
• ndarray-npy : to read and write npy files (numpy storage format)

Evolution

• parallelize operations : matrix multiplication, update weights, etc.
• implement convolutional layers (example). Convolutional layer applies filters to the input image to extract features. Convolution layer values are learned during training in the same way as the perceptron weights.

Run the project

Setup

1. Install rust
2. Extract dataset and pretrained model

unzip ./input/train.zip -d ./input/
unzip ./output/example_model_pretrained.zip -d ./output/example_model_pretrained

Run

Build and run the projet (model training + demo):

cargo run 

By default, the application:

1. load the input dataset
2. shuffle the dataset data
3. split the dataset into a training set and a test set
4. train the model on the train set
5. evaluate the model on the test set

Alternative use

The different stages of the process can also be run independently.

For example, you can:

• load a pretrained model
• skip the training
• skip the evaluation

Examples :

Run only the demo with a pretrained model :

cargo run -- --skip-training --test-rate 0 --input_model ./output/example_model_pretrained

Only evaluate a pretrained model :

cargo run -- --skip-training --test-rate 100 --input_model ./output/example_model_pretrained

Documentation :

cargo run -- --help

Test

Run test suite :

cargo test