Calculating the weights for an ADALINE using the Delta Rule

• Download source code - 20.5 KB
• Download compiled application - 9.23 KB

Introduction

This is actually one of the simplest C++ programs I have ever completed. There really isn't much to it. It won't be of interest to either accomplished C++ programmers, or to experts in Neural Networks. I hope however, it may be of some interest to people who are roughly at the same stage as me in learning at Neural Networks, i.e., pretty much at the start. It is also a fairly easy Visual C++ program to write. In other articles I have written, I have described how to build the application up with a fair amount of detail, starting with describing how to deal with the Application Wizard. In this article, I will assume the reader knows how to create an application shell using the Application Wizard, and how to place controls on a form and attach variables to those controls, and how to make an application handler function for a button control.

For those who aren't interested in the coding but would like to play with the calculator, the compiled application is available by clicking on the link above. It is easy to use. You enter your starting weights, training set, desired output, and value for eta, and press 'Go'. The program should stop running pretty quickly. The compiled program will run for one hundred passes, which, as far as I can tell, is fully adequate. If you scroll down the output box until the numbers in column 'y' (the output) start to correspond with the numbers in column 'd' (the desired output), then the values shown in the output box under 'w0', 'w1', and 'w2' on that line are the correct weights for the network. You can test this by replacing the 'Starting weights' with these values and pressing 'Go' again. This time, every value in 'y' should equal the corresponding value in 'd' right from the start of the output.

Background

I have been looking at Phil Picton's book 'Neural Networks', having had the book for some time and not getting through it as fast as I would like. It has only been in the last few days, however, that I had the bright idea of working through the iterative process used for finding the weights on an ADALINE. I wrote out the numbers on a piece of paper, and did the sums in my head between serving customers at the supermarket where I work. I felt pleased because I felt that I had got the process more or less right. However, the process wasn't fast enough for me to get satisfactory results. People in shops can be very demanding, you know. Besides, I suck at mental arithmetic. So, on a Saturday night, I set to and wrote this rather undemanding little program.

I am not going to try and describe the ADALINE here. I am only a learner. I will provide the appropriate Google search for the reader here - ADALINE Neural Network description. I am sure there is something useful out there. If not, then you could do worse than to buy Phil Picton's book.

Using the code

OK, very briefly, I set up an SDI using inheritance from CFormView. On the dialog, I added a number of text boxes for the input, and one for the output, plus one button to set everything going. All the code of any interest is in the event handler for the button CAdalineView::OnButtonGo() and one other member function I created called CAdalineView::adaline(...).

void CAdalineView::OnButtonGo() 
{
    // TODO: Add your control notification handler code here

    float d1, d2, d3, d4, eta;
    float y1, y2, y3, y4;
    float w10, w11, w12;
    float w20, w21, w22;
    float w30, w31, w32;
    float w40, w41, w42;
    float x10, x11, x12;
    float x20, x21, x22;
    float x30, x31, x32;
    float x40, x41, x42;
    
    float net1, net2, net3, net4;

    float dw10, dw11, dw12;
    float dw20, dw21, dw22;
    float dw30, dw31, dw32;
    float dw40, dw41, dw42;

    eta = atof(m_strEta);

    d1 = atof(m_strD1);
    d2 = atof(m_strD2);
    d3 = atof(m_strD3);
    d4 = atof(m_strD4);

    CString str="";
    m_strOutput="";


    ////Starting data

    w10 = atof(m_strW0);    
    w11 = atof(m_strW1);
    w12 = atof(m_strW2);
    int passes = 0;
    int matches = 0;

    x10 = atof(m_strX10);
    x11 = atof(m_strX11);
    x12 = atof(m_strX12);
    x20 = atof(m_strX20);
    x21 = atof(m_strX21);
    x22 = atof(m_strX22);
    x30 = atof(m_strX30);
    x31 = atof(m_strX31);
    x32 = atof(m_strX32);
    x40 = atof(m_strX40);
    x41 = atof(m_strX41);
    x42 = atof(m_strX42);

    while (passes <= 100){
        ///// input pattern 1


        adaline(x10, x11, x12, w10, w11, w12, d1, net1, y1,
                dw10, dw11, dw12, eta);

        //// change weights from pass1 to pass 2

        w20 = w10+dw10;
        w21 = w11+dw11;
        w22 = w12+dw12;

        ///// input pattern 2

        adaline(x20, x21, x22, w20, w21, w22, d2, net2, y2,
                dw20, dw21, dw22, eta);

        //// change weights from pass2 to pass 3

        w30 = w20+dw20;
        w31 = w21+dw21;
        w32 = w22+dw22;

        ///// input pattern 3

        adaline(x30, x31, x32, w30, w31, w32, d3, net3, y3,
                dw30, dw31, dw32, eta);

        //// change weights from pass3 to pass 4

        w40 = w30+dw30;
        w41 = w31+dw31;
        w42 = w32+dw32;

        ///// input pattern 4

        adaline(x40, x41, x42, w40, w41, w42, d4, net4, y4,
                dw40, dw41, dw42, eta);


        UpdateData(FALSE);


        //// change weights from pass4 to pass 1

        w10 = w40+dw40;
        w11 = w41+dw41;
        w12 = w42+dw42;


        passes++;
    }
}

So, this function takes the data from the edit boxes and handles the iteration process, passing the data for each of the four input patterns to the adaline function in turn. The weights are adjusted here, also. However, the values by which the weights are adjusted are calculated in the adaline function.

void CAdalineView::adaline(float x0, float x1, float x2, 
                            float w0, float w1, float w2,
                           float d, float net, float y, 
                           float &dw0, float &dw1, float &dw2, float eta)
{

    CString str = "";

    //Calculate 'net'
    net = x0*w0 + x1*w1 + x2*w2;

    //Calculate output value by passing 'net' through
    //a hard-limiter

    if (net >= 0)
        y = 1;
    else
        y = -1;
    
    //Here the 'Delta Rule' is applied

    dw0 = eta*x0*(d - net);
    dw1 = eta*x1*(d - net);
    dw2 = eta*x2*(d - net);

    //And the output is shuffled off to the appropriate text box
    //inputs

    str.Format("%.2f\t%.2f\t%.2f\t",x0,x1,x2);
    m_strOutput += str;
    //weights

    str.Format("%.2f\t%.2f\t%.2f\t",w0,w1,w2);
    m_strOutput += str;
    //d, net, y

    str.Format("%.2f\t%.2f\t%.2f\t",d,net,y);
    m_strOutput += str;
    //weight adjustments

    str.Format("%.2f\t%.2f\t%.2f\t\r\n",dw0,dw1,dw2);
    m_strOutput += str;

}

Note that dw1, dw2, and dw3 are passed 'by reference' (indicated by the ampersand prefix). It is both easier and more efficient to do this, allowing the variables to be assigned new values in the called function and have those new values instantly available in the calling function (because both functions are looking at the same memory locations). This precludes the various overheads incurred by passing the variables 'by value'. The other parameters are left as 'by value' passes since they are not required to provide information back to the calling function. If you are looking for more efficiency and want to keep with the principle of minimum privilege (i.e., the function being called can't interfere with the variable in the calling function) that 'by value' passes provide, then you could have all of the variables passed 'by reference', and declare all of them, other than the ones you specifically want to manipulate, in the calling function (i.e., dw0, dw1, as dw2) as 'const'.

The following snippet is where the Delta Rule is applied:

//Here the 'Delta Rule' is applied

dw0 = eta*x0*(d - net);
dw1 = eta*x1*(d - net);
dw2 = eta*x2*(d - net);

and here the output 'y' is calculated by passing the value in 'net' through a 'hard-limiter':

//Calculate output value by passing 'net' through

//a hard-limiter

if (net >= 0)
    y = 1;
else
    y = -1;

which, somehow, turns out to be a very friendly looking couple of lines of code to execute such a severe sounding function.

Points of interest

If I hadn't written this program, and carried on trying to do the calculations on a piece of paper, I would have been doing it till kingdom come, since I had been subtracting the weight changes (the dw values) from the weights (the w values). My program showed very efficiently that this led nowhere fast. After a moment or two of thinking about it, I changed the code (in the OnButtonGo function) to add the weight changes, and thankfully, it worked.

w20 = w10+dw10;
w21 = w11+dw11;
w22 = w12+dw12;