Simple deep neural networks tested with sine wave data trained using backpropagation 1 17 17 1 network to train a simple sine wave also plots the ideal and predicted in real time as it trains the network Be warned large networks with lots of layers is really really slow at training Some asserts scattered around to check basic array size aligment data let fp await fetch https cdn plot ly plotly 2 1 0 min js let ft await fp text var script document createElement script script src https cdn plot ly plotly 2 1 0 min js script innerHTML ft document head appendChild script const VARIANCE_W 0 5 const randomUniform min max return 0 123 Math random max min min const randomUniform min max Math random max min min const ru return randomUniform VARIANCE_W VARIANCE_W const layers 1 17 17 1 const maxLayerSize layers sort a b b a 0 const xordataset inputs 0 0 outputs 0 inputs 0 1 outputs 1 inputs 1 0 outputs 1 inputs 1 1 outputs 0 console assert xordataset 0 outputs length layers layers length 1 const weights Array layers length maxLayerSize maxLayerSize fill 0 const biases Array layers length maxLayerSize fill 0 const loutputs Array layers length maxLayerSize fill 0 const errors Array layers length maxLayerSize fill 0 const MAX_NEURONS_PER_LAYER maxLayerSize const NUM_LAYERS layers length for let i 0 i layers length 1 i for let k 0 k layers i i for let g 0 g layers i 1 g setWeight i k g ru function getBias layer neuron return biases layer MAX_NEURONS_PER_LAYER neuron function setBias layer neuron value biases layer MAX_NEURONS_PER_LAYER neuron value function setOutput layer neuron value loutputs layer MAX_NEURONS_PER_LAYER neuron value function getOutput layer neuron return loutputs layer MAX_NEURONS_PER_LAYER neuron function getWeight layer fromNeuron toNeuron return weights layer MAX_NEURONS_PER_LAYER MAX_NEURONS_PER_LAYER fromNeuron MAX_NEURONS_PER_LAYER toNeuron function setWeight layer fromNeuron toNeuron value weights layer MAX_NEURONS_PER_LAYER MAX_NEURONS_PER_LAYER fromNeuron MAX_NEURONS_PER_LAYER toNeuron value function setError layer neuron value errors layer MAX_NEURONS_PER_LAYER neuron value function getError layer neuron return errors layer MAX_NEURONS_PER_LAYER neuron let sigmoid x 1 0 1 0 Math exp x let sigmoidDerivative x x 1 x const relu x return Math max 0 0 x const reluDerivative x if x 0 0 return 1 0 return 0 0 const leakyRelu x alpha 0 01 return x 0 x alpha x const leakyReluDerivative x alpha 0 01 return x 0 1 alpha sigmoid leakyRelu sigmoidDerivative leakyReluDerivative let counter 0 const activate iin console assert iin length outputs 0 length console assert weights 0 length layers 0 console assert weights 0 0 length layers 1 console assert weights 1 length layers 1 console assert weights 1 0 length layers 2 counter for let i 0 i NUM_LAYERS i if i 0 for let k 0 k iin length k setOutput 0 k iin k else for let k 0 k layers i k var sum 0 0 for let b 0 b layers i 1 b sum getOutput i 1 b getWeight i 1 b k setOutput i k sigmoid sum getBias i k return getOutput NUM_LAYERS 1 0 const propagate target alpha 0 2 for let i NUM_LAYERS 1 i 0 i for let k 0 k layers i k if i NUM_LAYERS 1 let error target k getOutput i k sigmoidDerivative getOutput i k setError i k error else setError i k 0 0 for let g 0 g layers i 1 g let error getError i 1 g getWeight i k g sigmoidDerivative getOutput i k setError i k error for let i 0 i NUM_LAYERS i for let k 0 k layers i k if i NUM_LAYERS 1 for let g 0 g layers i 1 g var weight getWeight i k g weight alpha getOutput i k getError i 1 g setWeight i k g weight let bias getBias i k bias alpha getError i k setBias i k bias test the neural network using iteration loop xor dataset console log new Date for let epoch 0 epoch 10000 epoch let indexes Array from Array xordataset length keys indexes sort Math random 0 5 for let j of indexes activate xordataset j inputs propagate xordataset j outputs 0 2 if epoch 1000 0 let cost 0 for let j 0 j xordataset length j let o activate xordataset j inputs for let b 0 b xordataset j outputs length b cost Math pow xordataset j outputs b o b 2 cost 4 console log epoch epoch mean squared error cost for let i 0 i xordataset length i const result activate xordataset i inputs console log for input xordataset i inputs expected xordataset i outputs predicted result 0 toFixed 4 which is Math round result 0 xordataset i outputs 0 correct incorrect SINE WAVE let noSamples 50 let iteration 0 var trainingSet Function to shuffle data function shuffle array for let i array length 1 i 0 i const j Math floor Math random i 1 array i array j array j array i return array Function to create mini batches function createMiniBatches data batchSize let miniBatches for let i 0 i data length i batchSize miniBatches push data slice i i batchSize return miniBatches async function calcTotalError let totalError 0 for let tt 0 tt noSamples tt 1 nn value let amp await activate tt noSamples let val amp 0 2 0 1 0 ideal value let t 2 Math PI tt noSamples let mval Math sin t totalError totalError Math abs val mval let elem2 document getElementById error if elem2 undefined elem2 document createElement div elem2 id error document body appendChild elem2 elem2 innerHTML Error totalError iteration starts once the data has been loaded async function iterate iteration let elem1 document getElementById counter if elem1 undefined elem1 document createElement div elem1 id counter document body appendChild elem1 elem1 innerHTML Iteration iteration if iteration 10 0 only update graph every 5 iterations await plotData calcTotalError if trainingSet undefined console log creating dataset trainingSet for let tt 0 tt noSamples tt 1 let t 2 Math PI tt noSamples console assert t 0 t 1 let amp Math sin t 1 to 1 let ampNormalized amp 1 0 0 5 0 to 1 console assert ampNormalized 0 console assert ampNormalized 1 trainingSet push inputs tt noSamples outputs ampNormalized const NUM_EPOCHS 5 const BATCH_SIZE 20 for let epoch 0 epoch NUM_EPOCHS epoch let shuffledData shuffle trainingSet let miniBatches createMiniBatches shuffledData BATCH_SIZE for let miniBatch 0 miniBatch miniBatches length miniBatch let batches miniBatches miniBatch for let bat 0 bat batches length bat let datum batches bat await activate datum inputs await propagate datum outputs for let i 0 i trainingSet length i BATCH_SIZE let batch trainingSet slice i i BATCH_SIZE batch forEach datum activate datum inputs propagate datum outputs requestAnimationFrame iterate requestAnimationFrame iterate async function plotData plot trained data let x let y let y2 for let tt 0 tt noSamples tt 1 let t 2 Math PI tt noSamples let amp await activate tt noSamples let val amp 0 2 0 1 0 x push tt noSamples y push val y2 push Math sin t let plot1 document getElementById plot1 if plot1 undefined plot1 document createElement div plot1 id plot1 plot1 style width 600px plot1 style height 400px document body appendChild plot1 Plotly newPlot plot1 x y margin t 0 showSendToCloud true plot ideal data Plotly addTraces plot1 x y y2 end plotData deep neural network 1 time time index for input 17 17 hidden layers 1 amplitude
let bias getBias i k bias alpha getError i k setBias i k bias test the neural network using iteration loop xor dataset console log new Date for let epoch 0 epoch 10000 epoch let indexes Array from Array xordataset length keys indexes sort Math random 0 5 for let j of indexes activate xordataset j inputs propagate xordataset j outputs 0 2 if epoch 1000 0 let cost 0 for let j 0 j xordataset length j let o activate xordataset j inputs for let b 0 b xordataset j outputs length b cost Math pow xordataset j outputs b o b 2 cost 4 console log epoch epoch mean squared error cost for let i 0 i xordataset length i const result activate xordataset i inputs console log for input xordataset i inputs expected xordataset i outputs predicted result 0 toFixed 4 which is Math round result 0 xordataset i outputs 0 correct incorrect SINE WAVE let noSamples 50 let iteration 0 var trainingSet Function to shuffle data function shuffle array for let i array length 1 i 0 i const j Math floor Math random i 1 array i array j array j array i return array Function to create mini batches function createMiniBatches data batchSize let miniBatches for let i 0 i data length i batchSize miniBatches push data slice i i batchSize return miniBatches async function calcTotalError let totalError 0 for let tt 0 tt noSamples tt 1 nn value let amp await activate tt noSamples let val amp 0 2 0 1 0 ideal value let t 2 Math PI tt noSamples let mval Math sin t totalError totalError Math abs val mval let elem2 document getElementById error if elem2 undefined elem2 document createElement div elem2 id error document body appendChild elem2 elem2 innerHTML Error totalError iteration starts once the data has been loaded async function iterate iteration let elem1 document getElementById counter if elem1 undefined elem1 document createElement div elem1 id counter document body appendChild elem1 elem1 innerHTML Iteration iteration if iteration 10 0 only update graph every 5 iterations await plotData calcTotalError if trainingSet undefined console log creating dataset trainingSet for let tt 0 tt noSamples tt 1 let t 2 Math PI tt noSamples console assert t 0 t 1 let amp Math sin t 1 to 1 let ampNormalized amp 1 0 0 5 0 to 1 console assert ampNormalized 0 console assert ampNormalized 1 trainingSet push inputs tt noSamples outputs ampNormalized const NUM_EPOCHS 5 const BATCH_SIZE 20 for let epoch 0 epoch NUM_EPOCHS epoch let shuffledData shuffle trainingSet let miniBatches createMiniBatches shuffledData BATCH_SIZE for let miniBatch 0 miniBatch miniBatches length miniBatch let batches miniBatches miniBatch for let bat 0 bat batches length bat let datum batches bat await activate datum inputs await propagate datum outputs for let i 0 i trainingSet length i BATCH_SIZE let batch trainingSet slice i i BATCH_SIZE batch forEach datum activate datum inputs propagate datum outputs requestAnimationFrame iterate requestAnimationFrame iterate async function plotData plot trained data let x let y let y2 for let tt 0 tt noSamples tt 1 let t 2 Math PI tt noSamples let amp await activate tt noSamples let val amp 0 2 0 1 0 x push tt noSamples y push val y2 push Math sin t let plot1 document getElementById plot1 if plot1 undefined plot1 document createElement div plot1 id plot1 plot1 style width 600px plot1 style height 400px document body appendChild plot1 Plotly newPlot plot1 x y margin t 0 showSendToCloud true plot ideal data Plotly addTraces plot1 x y y2 end plotData deep neural network 1 time time index for input 17 17 hidden layers 1 amplitude