Simple neural networks tested with xor and back propagation const VARIANCE_W 0 5 const randomUniform min max Math random max min min let w11 randomUniform VARIANCE_W VARIANCE_W let w21 randomUniform VARIANCE_W VARIANCE_W let b1 0 let w12 randomUniform VARIANCE_W VARIANCE_W let w22 randomUniform VARIANCE_W VARIANCE_W let b2 0 let w13 randomUniform VARIANCE_W VARIANCE_W let w23 randomUniform VARIANCE_W VARIANCE_W let b3 0 let o1 randomUniform VARIANCE_W VARIANCE_W let o2 randomUniform VARIANCE_W VARIANCE_W let o3 randomUniform VARIANCE_W VARIANCE_W let ob 0 const sigmoid x 1 0 1 0 Math exp x const sigmoidPrime x x 1 x const predict i1 i2 let s1 w11 i1 w21 i2 b1 s1 sigmoid s1 let s2 w12 i1 w22 i2 b2 s2 sigmoid s2 let s3 w13 i1 w23 i2 b3 s3 sigmoid s3 let output s1 o1 s2 o2 s3 o3 ob output sigmoid output return output const learn i1 i2 target alpha 0 2 let s1 w11 i1 w21 i2 b1 s1 sigmoid s1 let s2 w12 i1 w22 i2 b2 s2 sigmoid s2 let s3 w13 i1 w23 i2 b3 s3 sigmoid s3 let output s1 o1 s2 o2 s3 o3 ob output sigmoid output let error target output let derror error sigmoidPrime output let ds1 derror o1 sigmoidPrime s1 let ds2 derror o2 sigmoidPrime s2 let ds3 derror o3 sigmoidPrime s3 o1 alpha s1 derror o2 alpha s2 derror o3 alpha s3 derror ob alpha derror w11 alpha i1 ds1 w21 alpha i2 ds1 b1 alpha ds1 w12 alpha i1 ds2 w22 alpha i2 ds2 b2 alpha ds2 w13 alpha i1 ds3 w23 alpha i2 ds3 b3 alpha ds3 const INPUTS 0 0 0 1 1 0 1 1 const OUTPUTS 0 1 1 0 for let epoch 1 epoch 10000 epoch let indexes 0 1 2 3 indexes sort Math random 0 5 for let j of indexes learn INPUTS j 0 INPUTS j 1 OUTPUTS j 0 0 2 if epoch 1000 0 let cost 0 for let j 0 j 4 j let o predict INPUTS j 0 INPUTS j 1 cost Math pow OUTPUTS j 0 o 2 cost 4 console log epoch epoch mean squared error cost for let i 0 i 4 i let result predict INPUTS i 0 INPUTS i 1 console log for input INPUTS i expected OUTPUTS i 0 predicted result toFixed 4 which is Math round result OUTPUTS i 0 correct incorrect The variables are stored in array and the for loops more resemble the matrix operations With HIDDEN 3 it behaves the same as the first script const VARIANCE_W 0 5 const INPUTS 2 const HIDDEN 3 const OUTPUTS 1 const randomUniform min max Math random max min min let hidden_weights for let i 0 i HIDDEN i let weights for let j 0 j INPUTS j weights push randomUniform VARIANCE_W VARIANCE_W hidden_weights push weights let hidden_bias Array HIDDEN fill 0 let output_weights for let i 0 i OUTPUTS i let weights for let j 0 j HIDDEN j weights push randomUniform VARIANCE_W VARIANCE_W output_weights push weights let output_bias Array OUTPUTS fill 0 const sigmoid x 1 0 1 0 Math exp x const sigmoidPrime x x 1 x const predict inputs let hiddens for let i 0 i HIDDEN i let hidden 0 for let j 0 j INPUTS j hidden hidden_weights i j inputs j hidden sigmoid hidden hidden_bias i hiddens push hidden let outputs for let i 0 i OUTPUTS i let output 0 for let j 0 j HIDDEN j output output_weights i j hiddens j output sigmoid output output_bias i outputs push output return outputs 0 const learn inputs targets alpha 0 1 let hiddens for let i 0 i HIDDEN i let hidden 0 for let j 0 j INPUTS j hidden hidden_weights i j inputs j hidden sigmoid hidden hidden_bias i hiddens push hidden let outputs for let i 0 i OUTPUTS i let output 0 for let j 0 j HIDDEN j output output_weights i j hiddens j output sigmoid output output_bias i outputs push output let errors for let i 0 i OUTPUTS i let error targets i outputs i errors push error let derrors for let i 0 i OUTPUTS i let derror errors i sigmoidPrime outputs i derrors push derror let ds Array HIDDEN fill 0 for let i 0 i OUTPUTS i for let j 0 j HIDDEN j ds j derrors i output_weights i j sigmoidPrime hiddens j for let i 0 i OUTPUTS i for let j 0 j HIDDEN j output_weights i j alpha hiddens j derrors i output_bias i alpha derrors i for let i 0 i HIDDEN i for let j 0 j INPUTS j hidden_weights i j alpha inputs j ds i hidden_bias i alpha ds i const inputs 0 0 0 1 1 0 1 1 const outputs 0 1 1 0 for let i 0 i 10000 i let indexes 0 1 2 3 indexes sort Math random 0 5 for let j of indexes learn inputs j outputs j 0 2 if i 1 1000 0 let cost 0 for let j 0 j 4 j let o predict inputs j cost Math pow outputs j 0 o 2 cost 4 console log i 1 mean squared error cost for let i 0 i 4 i let result predict inputs i console log for input inputs i expected outputs i 0 predicted result toFixed 4 which is Math round result outputs i 0 correct incorrect Add momentum next to the weights Here instead of updating weights directly we update the momentum and then the weights according to the momentum const VARIANCE_W 0 5 const INPUTS 2 const HIDDEN 3 const OUTPUTS 1 const randomUniform min max Math random max min min let hidden_weights let hidden_momentum for let i 0 i HIDDEN i let weights let momentum for let j 0 j INPUTS j weights push randomUniform VARIANCE_W VARIANCE_W momentum push 0 hidden_weights push weights hidden_momentum push momentum let hidden_bias Array HIDDEN fill 0 let output_weights let output_momentum for let i 0 i OUTPUTS i let weights let momentum for let j 0 j HIDDEN j weights push randomUniform VARIANCE_W VARIANCE_W momentum push 0 output_weights push weights output_momentum push momentum let output_bias Array OUTPUTS fill 0 const sigmoid x 1 0 1 0 Math exp x const sigmoidPrime x x 1 x const tanh x Math tanh x const tanhPrime x 1 x x const predict inputs let hiddens for let i 0 i HIDDEN i let hidden 0 for let j 0 j INPUTS j hidden hidden_weights i j inputs j hidden tanh hidden hidden_bias i hiddens push hidden let outputs for let i 0 i OUTPUTS i let output 0 for let j 0 j HIDDEN j output output_weights i j hiddens j output sigmoid output output_bias i outputs push output return outputs 0 const learn inputs targets alpha 0 1 lambd 0 8 let hiddens for let i 0 i HIDDEN i let hidden 0 for let j 0 j INPUTS j hidden hidden_weights i j inputs j hidden tanh hidden hidden_bias i hiddens push hidden let outputs for let i 0 i OUTPUTS i let output 0 for let j 0 j HIDDEN j output output_weights i j hiddens j output sigmoid output output_bias i outputs push output let errors for let i 0 i OUTPUTS i let error targets i outputs i errors push error let derrors for let i 0 i OUTPUTS i let derror errors i sigmoidPrime outputs i derrors push derror let ds Array HIDDEN fill 0 for let i 0 i OUTPUTS i for let j 0 j HIDDEN j ds j derrors i output_weights i j tanhPrime hiddens j for let i 0 i OUTPUTS i for let j 0 j HIDDEN j output_momentum i j lambd output_momentum i j alpha hiddens j derrors i output_weights i j output_momentum i j output_bias i alpha derrors i for let i 0 i HIDDEN i for let j 0 j INPUTS j hidden_momentum i j lambd hidden_momentum i j alpha inputs j ds i hidden_weights i j hidden_momentum i j hidden_bias i alpha ds i const inputs 0 0 0 1 1 0 1 1 const outputs 0 1 1 0 for let i 0 i 10000 i let indexes 0 1 2 3 indexes sort Math random 0 5 for let j of indexes learn inputs j outputs j 0 2 0 8 if i 1 1000 0 let cost 0 for let j 0 j 4 j let o predict inputs j cost Math pow outputs j 0 o 2 cost 4 console log i 1 mean squared error cost for let i 0 i 4 i let result predict inputs i console log for input inputs i expected outputs i 0 predicted result toFixed 4 which is Math round result outputs i 0 correct incorrect
j 0 j HIDDEN j output_weights i j alpha hiddens j derrors i output_bias i alpha derrors i for let i 0 i HIDDEN i for let j 0 j INPUTS j hidden_weights i j alpha inputs j ds i hidden_bias i alpha ds i const inputs 0 0 0 1 1 0 1 1 const outputs 0 1 1 0 for let i 0 i 10000 i let indexes 0 1 2 3 indexes sort Math random 0 5 for let j of indexes learn inputs j outputs j 0 2 if i 1 1000 0 let cost 0 for let j 0 j 4 j let o predict inputs j cost Math pow outputs j 0 o 2 cost 4 console log i 1 mean squared error cost for let i 0 i 4 i let result predict inputs i console log for input inputs i expected outputs i 0 predicted result toFixed 4 which is Math round result outputs i 0 correct incorrect Add momentum next to the weights Here instead of updating weights directly we update the momentum and then the weights according to the momentum const VARIANCE_W 0 5 const INPUTS 2 const HIDDEN 3 const OUTPUTS 1 const randomUniform min max Math random max min min let hidden_weights let hidden_momentum for let i 0 i HIDDEN i let weights let momentum for let j 0 j INPUTS j weights push randomUniform VARIANCE_W VARIANCE_W momentum push 0 hidden_weights push weights hidden_momentum push momentum let hidden_bias Array HIDDEN fill 0 let output_weights let output_momentum for let i 0 i OUTPUTS i let weights let momentum for let j 0 j HIDDEN j weights push randomUniform VARIANCE_W VARIANCE_W momentum push 0 output_weights push weights output_momentum push momentum let output_bias Array OUTPUTS fill 0 const sigmoid x 1 0 1 0 Math exp x const sigmoidPrime x x 1 x const tanh x Math tanh x const tanhPrime x 1 x x const predict inputs let hiddens for let i 0 i HIDDEN i let hidden 0 for let j 0 j INPUTS j hidden hidden_weights i j inputs j hidden tanh hidden hidden_bias i hiddens push hidden let outputs for let i 0 i OUTPUTS i let output 0 for let j 0 j HIDDEN j output output_weights i j hiddens j output sigmoid output output_bias i outputs push output return outputs 0 const learn inputs targets alpha 0 1 lambd 0 8 let hiddens for let i 0 i HIDDEN i let hidden 0 for let j 0 j INPUTS j hidden hidden_weights i j inputs j hidden tanh hidden hidden_bias i hiddens push hidden let outputs for let i 0 i OUTPUTS i let output 0 for let j 0 j HIDDEN j output output_weights i j hiddens j output sigmoid output output_bias i outputs push output let errors for let i 0 i OUTPUTS i let error targets i outputs i errors push error let derrors for let i 0 i OUTPUTS i let derror errors i sigmoidPrime outputs i derrors push derror let ds Array HIDDEN fill 0 for let i 0 i OUTPUTS i for let j 0 j HIDDEN j ds j derrors i output_weights i j tanhPrime hiddens j for let i 0 i OUTPUTS i for let j 0 j HIDDEN j output_momentum i j lambd output_momentum i j alpha hiddens j derrors i output_weights i j output_momentum i j output_bias i alpha derrors i for let i 0 i HIDDEN i for let j 0 j INPUTS j hidden_momentum i j lambd hidden_momentum i j alpha inputs j ds i hidden_weights i j hidden_momentum i j hidden_bias i alpha ds i const inputs 0 0 0 1 1 0 1 1 const outputs 0 1 1 0 for let i 0 i 10000 i let indexes 0 1 2 3 indexes sort Math random 0 5 for let j of indexes learn inputs j outputs j 0 2 0 8 if i 1 1000 0 let cost 0 for let j 0 j 4 j let o predict inputs j cost Math pow outputs j 0 o 2 cost 4 console log i 1 mean squared error cost for let i 0 i 4 i let result predict inputs i console log for input inputs i expected outputs i 0 predicted result toFixed 4 which is Math round result outputs i 0 correct incorrect