# NeuralNetworks

**Repository Path**: MeiJuice/neural-networks

## Basic Information

- **Project Name**: NeuralNetworks
- **Description**: Completely Connected Recursive Neural Networks (back propagation)，C++ Construction
- **Primary Language**: C++
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 2
- **Forks**: 0
- **Created**: 2021-01-03
- **Last Updated**: 2024-02-06

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Neural Networks

An implementation of feedforward neural networks in C++. NeuralNetwork class can be initiated with a desired architecture, trained on a set of data and then used on real examples.

This implementation uses neural networks for categorization problems. It outputs a number from 0 to 1 for each category, representing a confidence for that category. Sigmoid function is used as activation function and gradient descent as optimizer. Regularization is used as a measure against overfitting.

Also included is an example usage for analyzing the MNIST database. MNIST is a popular dataset of handwritten digits. You can learn more about it here: http://yann.lecun.com/exdb/mnist

## How to use

All needed files are under *source/neural_networks*. Include these in your project. What you will get are two classes: Matrix (for linear algebra calculations) and NeuralNetwork (which you will use).

NeuralNetwork can then be initialized and trained like this:

	NeuralNetwork *NN = new NeuralNetwork(inputLayerSize, vectorOfHiddenLayerSizes, outputCategoryCount);
		
	shuffleTrainSamplesSet();

	for (int i = 0; i < numberOfTrainEpochs; ++i) {
		for (TrainSample trainSample : trainSet) {
			for (int j = 0; j < inputLayerSize; ++j)
				NN->setInput(j, trainSample.inputVector[j]);
			
			// output labels as bool values
			NN->backpropagation(trainSample.correctOutputVector);
		}
		
		NN->gradientDescentStep();
	}

After training, use them on real or test samples like this:

	for (int i = 0; i < inputLayerSize; ++i)
		NN->setInput(i, sampleInputVector[i]);
	NN->feedforward();
	
	for (int i = 0; i < outputCategoryCount; ++i)
		categoryConfidence[i] = NN->getOutput(i);


## MNIST example

Under *source/example_usage* is a program which uses this implementation of neural networks on MNIST database (files in *MNIST* folder). This program will read train and test sets, train a neural network and test and print its accuracy achieved on the test set.

# Fast Gradient Sign Method
In the white box environment, by calculating the derivative of the model to the input, and then using the sign function to get the specific gradient direction, and then multiplying by a step, the "disturbance" obtained is added to the original input to get the sample under FGSM attack.

You can learn more about it here: https://arxiv.org/abs/1412.6572

## FGSM sample code
Under *source/GenerativeAdversaria/FGSM* folder, the anti sample attack generation method is placed and stuck in the neural network class.

After training the model, you can create new objects to implement the FGSM method to generate confrontation samples

Will improve the file reading and writing method over time.

## Generate results
The classification accuracy can be reduced from 97.5% to 9.5%. For the image generated by the gradient rising method of pixels, considering the distortion of the image will be higher, but the effect is more obvious

# Adversarial Transformation Networks
In many existing methods, the gradient information attack method accounts for the vast majority, so this paper proposes another method: training a depth network, taking the original image as the input, and outputting it as the counter sample.

You can learn more about it here: https://arxiv.org/abs/1703.09387

## ATN function code
On the basis of the source code, the ATN suffix is added to represent a new class of training model, in which the relevant methods for training ATN model are collected.

## Training method
Lx (visual loss function) and lx (classification loss function) are used to train the gradient descent of ATN network, and the balance of the two loss functions is adjusted through beta.

The gradient descent algorithm is similar to BP network.

## Optimization and Normalization
We clearly know that the output of ATN network is a confrontation sample image, but according to the original training method, the final result will be the activation value rather than the gray value, because the activation function will be used in the forward propagation.

In order to solve this problem, we should retrain a model whose output layer is the normalized training samples. Secondly, in the process of training ATN network, we should first normalize the samples before training.

## Generate results
Because the super parameter is not consistent with the paper, using *Beta = 0.01* in the paper can achieve very good results, and the classification attack can even reach 100%, but it also directly leads to the whole picture can not distinguish the label.

So the selection of the *beta* value of BP neural network is still in the experiment.