Introduction
This article is going to explain how you can implement an easy to use neuronal network with the example of character recognition.
You need 3 things for a neuronal network:
- A trainingset (known answers for a given problem)
- A neuronal network
- A way to tell the network it improved
This artificial network uses an evolutionary algorithm for training.
Here is the shortest possible example to train a new neuronal network:
First of all, add a reference to Synapse.dll.
That's all.
using Brains;
Brain ANN = new Brain(5, 4, 3);
ANN.SetInput(List<double> traindata);
while (ANN.score<10)
{
ANN.score=grade(ANN.Outputs);
}
Bitmap finish=ANN.ToBitmap();
This network would have 5 inputs 4 outputs and 3 hidden layers. It is almost similar to the first picture.
Of course, you would have to define the grade function. This is just a double number which tells the network if you like what it is doing or not.
As a user, you don't see the training algorithm and all the magic happens at the setter of ANN.score
The setter of the score value is the interface for a evolutionary algorithm which tries to always get a higher score than before.
In a nutshell, this works with:
- mutation
- selection
- reproduction
Notice: The user sets the score by his own rules so if the real output is closer to the desired output, the score should be a little bit higher.
Background
There are several things you need to know about neuronal networks:
The most important thing is a neuron. This is a construct which has many inputs and only one output of type double.
To use neurons in a useful way, they are connected with each other in many layers. Most of them are hidden and only the output and the inputlayer are visible from the "outside".
If you look at the first picture, you can see that every circle is a neuron.
The inputs of any given neuron (except inputlayer) would be all outputs of all the neurons of the layer before. The outputs of the inputlayer are the members of the list the user provides.
This project uses arctan(x)/pi as a transferfunction (continuous function which normalizes all rational values between -1 and 1).
The output formula for each neuron is as follows:
Output=Tranferfuction(sum(inputs*weights))
The weights of each neuron, in this case called genome is the key factor for a working neuronal network. Each neuron has one double value stored for each neuron connected to it. This factor can be between -4 and +4 in this case. This allows cancellation in the sum of all outputs. The neuronal network keeps a dictionary in the background to improve speed.
By setting the weights to certain (unknown) values, the neuronal network can "learn" the relation between an input and an output.
To actually train a neuronal network, all combinations of all weights would have to be tested to find a global optimum. It is not possible to know you have found the best solution.
Although the training algorithm tries to achieve a global optimum finding an optimal solution cannot be guaranteed.
The simplest implementation of a neuron would be:
class neuron
{ neuron
{
double output
{
get {tranferfunction(sum(previouslayer));}
private set{}
}
}
That's of course a simplification. The real code looks more like this:
double sum()
{
double back = 0;
for (int i = 0; i < previous.Count; i++) { back += previous[i].output * genome[i]; }
if (Parent.Transferfunction==Transferfunction.Stepfunction)
{
back = back / previous.Count;
}
back = transfer(back);
return back;
}
double ret = double.PositiveInfinity;
public double output
{
get
{
if (!double.IsInfinity(ret)) { return ret; }
else {
double outp;
if (Parent.antworten.ContainsKey(name) == false)
{
outp = sum();
lock (Parent) { Parent.antworten.Add(name, outp); }
return outp;
}
else { Parent.antworten.TryGetValue(name, out outp); return outp; } }
}
set { ret = value; }
}
For great reads about neuronal networks, please check out:
Using the Neuronal Net
This example project is going to show how to train the network to differentiate between 3 letters: A B and C.
This means we have a Bitmap as an input and 3 outputs. Each output is a number between 0 and 1.
First thing you have to do is to think about a way to turn a bitmap into a list of 0s and 1s:
Example pictures below:
apply_image(Bitmap image)
{
List<double>input = new List<double>();
brightness = image.GetPixel(x, y).GetBrightness();
if (brightness > 0.5) { input.Add(1); }
else input.Add(0);
}
If you have a picture with 320x120 resolution, then there would be too many inputs for the network.
Instead, you only take every 20th pixel or something like that. This is sufficient.
The code below contains all information needed to train a neuronal network.
The uncertainty of a neuronal network is the number and the size of hidden layers.
General rule for perceptrons is that they don't need hidden layers at all.
If you want to achieve higher functionality such as the XOR problem or multiplication, you will need more hidden layers.
Of course, training a neuronal network for multiplication would be a waste of time, because multiplication is an easy task for every programmer.
Notice: The Score property changes the existing network a little bit to perform better next time.
It keeps an internal list of configurations and corresponding scores to generate better configurations.
NET.StopTraining(); gets the best configuration so far.
A perfect score for B would be 0 1 0.
Bitmap A = new Bitmap(@"A.bmp");
Bitmap B = new Bitmap(@"B.bmp");
Bitmap C = new Bitmap(@"C.bmp");
Bitmap T1 = new Bitmap(@"TEST1.bmp");
Bitmap T2 = new Bitmap(@"TEST2.bmp");
Brain NET = new Brain(100, 3,0);
NET.Trainmode = Trainmode.Fast;
NET.Transferfunction = Transferfunction.Arctan;
NET.Outputmax = 1;
var inputs = apply_image(A);
double score = 0;
while (NET.Score<2.9)
{
score = 0;
inputs = apply_image(A);
NET.SetInput(inputs);
score += NET[0];
score -= NET[1];
score -= NET[2];
inputs = apply_image(B);
NET.SetInput(inputs);
score -= NET[0];
score += NET[1];
score -= NET[2];
inputs = apply_image(C);
NET.SetInput(inputs);
score -= NET[0];
score -= NET[1];
score += NET[2];
NET.Score = score;
}
NET.StopTraining();
The neuronal network should be able to correctly tell that case 2 is still an A (smaller not in the center and some artifacts).
The final output for the above testcases were as follows:
- A: 0.98 0.02 0.03
- A bad: 0.86 0.13 0.09
- C: 0.02 0.01 0.98
This tells us the training has succeeded and the network actually "knows" the difference between A, B and C!
To test the code yourself, please check out the demo project.
The Learning Algorithm
Each neuron has a set of weights called the genome. The whole network can be seen as a list of doubles because only each weight of each neuron is important.
If you train the network only this list has to be changed.
First thing you have to do is keep a list of all scores and genomes. For this, it is basically a list of doubles and the score. This snippet happens at the setter of score, because the user is resposible for the score and the trainingset.
list<genome> max;
if (value > max.score)
{
if (Trainmode == Trainmode.Fast)
{
if (best.Count == 0) {best.Add(new Genome(this));}
var max = best.MaxBy(x => x.score);
if (value > max.score) { score = value; Generation++;
best.Add(new Genome(this));
}
else { max.SetGenesTo(this); this.mutate(); }
}
Out of the list of good genomes, the best one gets chosen. After that, some weights (2%) are randomly changed.
This is called mutation and selectation. Reproduction is implied by copying the best genome.
The real implementation is more complicated because it treats more genomes as the best one to avoid local minima.
You can choose if you want to learn fast or try to avoid not optimal solutions. For this, I implemented the trainmode property.
You can set it to fast or quality. The first one always picks the best genome and develops it further. The second trainmode trains 10 genomes and then picks the best one.
After that, it again develops 10 genomes on the basis of the best one. This way, its training is much slower but you can achieve better results.
If you want to learn more about other training methods, please check the links in the background section!!
Points of Interest
<pmost>
During my coding, I always wanted to see what the network is doing. That's why I have implemented mynet.ToBitmap(); and mynet.ShowGraph(); which convert all inputs and parameters into one nice picture.
Most people think of neuronal networks as a form of artificial intelligence. A simple feed forward network as used in this article is not intelligent. You can rather think of an neuronal network as a mathematical formula. You do not know the formula but you do know the answer to some input parameters (trainingset). The learning algorithm alters the right parameters in the matrixmultiplications and thus finds a general formula for your training set.
