Teaching a Neural Net: Bipolar XOR -
i'm trying to teach neural net of 2 inputs, 4 hidden nodes (all in same layer) , 1 output node. binary representation works fine, have problems bipolar. can't figure out why, total error converge same number around 2.xx. sigmoid 2/(1+ exp(-x)) - 1. perhaps i'm sigmoiding in wrong place. example calculate output error should comparing sigmoided output expected value or sigmoided expected value?
i following website here: http://galaxy.agh.edu.pl/~vlsi/ai/backp_t_en/backprop.html , use different functions instructed use. when did try implement functions still ran same problem. either way stuck half time @ same number (a different number different implementations). please tell me if have made mistake in code somewhere or if normal (i don't see how be). momentum set 0. common 0 momentum problem? error functions supposed using are:
if ui output unit
error(i) = (ci - ui ) * f'(si )
if ui hidden unit
error(i) = error(output) * weight(i output) * f'(si)
public double sigmoid( double x ) { double fbipolar, fbinary, temp; temp = (1 + math.exp(-x)); fbipolar = (2 / temp) - 1; fbinary = 1 / temp; if(bipolar){ return fbipolar; }else{ return fbinary; } } // initialize weights random values. private void initializeweights(double neg, double pos) { for(int = 0; < numinputs + 1; i++){ for(int j = 0; j < numhiddenneurons; j++){ inputweights[i][j] = math.random() - pos; if(inputweights[i][j] < neg || inputweights[i][j] > pos){ print("error "); print(inputweights[i][j]); } } } for(int = 0; < numhiddenneurons + 1; i++){ hiddenweights[i] = math.random() - pos; if(hiddenweights[i] < neg || hiddenweights[i] > pos){ print("error "); print(hiddenweights[i]); } } } // computes output of nn without training. i.e. forward pass public double outputfor ( double[] arginputvector ) { for(int = 0; < numinputs; i++){ inputs[i] = arginputvector[i]; } double weightedsum = 0; for(int = 0; < numhiddenneurons; i++){ weightedsum = 0; for(int j = 0; j < numinputs + 1; j++){ weightedsum += inputweights[j][i] * inputs[j]; } hiddenactivation[i] = sigmoid(weightedsum); } weightedsum = 0; for(int j = 0; j < numhiddenneurons + 1; j++){ weightedsum += (hiddenactivation[j] * hiddenweights[j]); } return sigmoid(weightedsum); } //computes derivative of f public static double fprime(double u){ double fbipolar, fbinary; fbipolar = 0.5 * (1 - math.pow(u,2)); fbinary = u * (1 - u); if(bipolar){ return fbipolar; }else{ return fbinary; } } // method used update weights of neural net. public double train ( double [] arginputvector, double argtargetoutput ){ double output = outputfor(arginputvector); double lastdelta; double outputerror = (argtargetoutput - output) * fprime(output); if(outputerror != 0){ for(int = 0; < numhiddenneurons + 1; i++){ hiddenerror[i] = hiddenweights[i] * outputerror * fprime(hiddenactivation[i]); deltahiddenweights[i] = learningrate * outputerror * hiddenactivation[i] + (momentum * lastdelta); hiddenweights[i] += deltahiddenweights[i]; } for(int in = 0; in < numinputs + 1; in++){ for(int hid = 0; hid < numhiddenneurons; hid++){ lastdelta = deltainputweights[in][hid]; deltainputweights[in][hid] = learningrate * hiddenerror[hid] * inputs[in] + (momentum * lastdelta); inputweights[in][hid] += deltainputweights[in][hid]; } } } return 0.5 * (argtargetoutput - output) * (argtargetoutput - output); }
general coding comments:
initializeweights(-1.0, 1.0);
may not initial values expecting.
initializeweights should have:
inputweights[i][j] = math.random() * (pos - neg) + neg; // ... hiddenweights[i] = (math.random() * (pos - neg)) + neg;
instead of:
math.random() - pos;
so works:
initializeweights(0.0, 1.0);
and gives initial values between 0.0 , 1.0 rather between -1.0 , 0.0.
lastdelta
used before declared:
deltahiddenweights[i] = learningrate * outputerror * hiddenactivation[i] + (momentum * lastdelta);
i'm not sure if + 1
on numinputs + 1
, numhiddenneurons + 1
necessary.
remember watch out rounding of ints: 5/2 = 2, not 2.5! use 5.0/2.0 instead. in general, add .0 in code when output should double.
most importantly, have trained neuralnet long enough?
try running numinputs = 2, numhiddenneurons = 4, learningrate = 0.9, , train 1,000 or 10,000 times.
using numhiddenneurons = 2 "stuck" when trying solve xor problem.
Comments
Post a Comment