backprop1

Neural network - backpropagation: description

In [4]:

from IPython.display import Image
Image(filename = "nn1.png", width=200)

Out[4]:

Having the following simple training set (consisting of only one record data):

$$({ x }_{ 1 },{ x }_{ 2 },...,{ x }_{ n },target)$$

the scope of the supervised learning process of the above neural network is to simply match (as well as possible) the output to the given target by adjusting the weights of the perceptrons:

In [7]:

Image(filename = "blackbox.png", width=300)

Out[7]:

This is the internal mechanism of the "Neural net" black-box above:

$Sigmoid\quad function:\quad \sigma (x)=\frac { 1 }{ 1+{ e }^{ -x } } $

${ zh }_{ 1 }={ w }_{ 1 }{ x }_{ 1 }+{ w }_{ 3 }{ x }_{ 2 }+{ w }_{ 5 }{ x }_{ 3 }+{ b }_{ 1 }$

${ zh }_{ 2 }={ w }_{ 2 }{ x }_{ 1 }+{ w }_{ 4 }{ x }_{ 2 }+{ w }_{ 6 }{ x }_{ 3 }+{ b }_{ 1 }$

${ h }_{ 1 }=\sigma ({ zh }_{ 1 })$

${ h }_{ 2 }=\sigma ({ zh }_{ 2 })$

$zo_{ 1 }={ w }_{ 7 }h_{ 1 }+{ w }_{ 9 }{ h }_{ 2 }+{ b }_{ 2 }$

$zo_{ 2 }={ w }_{ 8 }{ h }_{ 1 }+{ w }_{ 10 }{ h }_{ 2 }+{ b }_{ 2 }$

$o_{ 1 }=\sigma ({ zo }_{ 1 })$

$o_{ 2 }=\sigma ({ zo }_{ 2 })$

$E=\frac { 1 }{ 2 } [{ ({ o }_{ 1 }-{ t }_{ 1 }) }^{ 2 }+{ ({ o }_{ 2 }-{ t }_{ 2 }) }^{ 2 }]$

E is the error function which evaluates the amount of "success" of the output.

Having all that, we need to calculate all the gradients of error (how much the error is "generated" by each of the weight) : the derivatives of the error with respect to each of the weights (including biases):

$$\frac { dE }{ { dw }_{ i } } ,\frac { dE }{ { db }_{ i } } $$

Then, the process of training the net consist in a number of loops containing two stages:

1)"forward propagation:"

process the output(using some initial random weights)
evaluate error
evaluate the values of the gradients

2)"backward propagation:"

adjusting the weights according to the formula:

$${ w }_{ i }={ w }_{ i }-\frac { dE }{ { dw }_{ i } } \cdot learn\_ rate$$

$${ b }_{ i }={ b }_{ i }-\frac { dE }{ { db }_{ i } } \cdot learn\_ rate$$

given the fact that:

if $\frac { dE }{ { dw }_{ i } }>0$, E is increasing if w increase, so, to decrease E, we must decrease w
if $\frac { dE }{ { dw }_{ i } }<0$, E is decreasing if w increase, so, to decrease E, we must increase w
if $\frac { dE }{ { dw }_{ i } }=0$, E riches a local minimum

Necessary math preliminaries for gradients evaluation:

1)Derivative of sigmoid function:

$\sigma '(x)=\frac { d }{ dx } \frac { 1 }{ 1+{ e }^{ -x } } =\frac { d }{ dx } { (1+{ e }^{ -x }) }^{ -1 }=-{ (1+{ e }^{ -x }) }^{ -2 }\cdot ({ -e }^{ -x })=\frac { { e }^{ -x } }{ { (1+{ e }^{ -x }) }^{ 2 } } =\frac { 1 }{ 1+{ e }^{ -x } } \cdot \frac { { e }^{ -x } }{ 1+{ e }^{ -x } } \Rightarrow \\ \sigma '(x)=\sigma (x)(1-\sigma (x))\\ $

2)Chain rule:

$(f(g(x)))'\quad =\quad f'(g(x))\cdot g'(x)$

$\frac { d }{ dx } f(g(x))=\frac { d\quad f(g(x)) }{ d\quad g(x) } \frac { d\quad g(x) }{ dx }$

$which\quad could\quad be\quad simplified\quad to\quad the\quad following\quad conventional\quad notation$

$\frac { d }{ dx } f(g(x))=\frac { df }{ dg } \frac { dg }{ dx }$

$or$

$\frac { d }{ dx } (f\circ g)(x)=\frac { df }{ dg } \frac { dg }{ dx }$

$and\quad generalized:$

$\frac { d }{ dx } (f\circ g\circ h\circ i...)(x)=\frac { df }{ dg } \frac { dg }{ dh } \frac { dh }{ di } \frac { di }{ ... } ...\frac { ... }{ dx }$

3)Partial differentiation: excepting the variable with respect which we differentiate, all other variables of the expression are considered constants so their derivatives are 0.

$\frac { dE }{ { dw }_{ 7 } } =\frac { 1 }{ 2 } 2({ o }_{ 1 }{ -t }_{ 1 })\frac { { do }_{ 1 } }{ { dw }_{ 7 } } +\frac { 1 }{ 2 } 2({ o }_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { dw }_{ 7 } } =$

$=({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { dw }_{ 7 } } +({ o }_{ 2 }-{ t }_{ 2 })\cdot 0=$

$=({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { dw }_{ 7 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ h }_{ 1 }$

All the gradients will be:

$\frac { dE }{ d{ w }_{ 7 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { dw }_{ 7 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { dw }_{ 7 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ h }_{ 1 }$

$\frac { dE }{ d{ w }_{ 8 } } =({ o }_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { dw }_{ 8 } } =({ o }_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { dzo }_{ 2 } } \frac { { dzo }_{ 2 } }{ { dw }_{ 8 } } =({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ h }_{ 1 }$

$\frac { dE }{ d{ w }_{ 9 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { dw }_{ 9 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { dw }_{ 9 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ h }_{ 2 }$

$\frac { dE }{ d{ w }_{ 10 } } =({ o }_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { dw }_{ 10 } } =({ o }_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { dzo }_{ 2 } } \frac { { dzo }_{ 2 } }{ { dw }_{ 10 } } =({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ h }_{ 2 }$

$\frac { dE }{ { dw }_{ 1 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { d }{ { dw }_{ 1 } } { o }_{ 1 }+({ o }_{ 2 }-{ t }_{ 2 })\frac { d }{ { dw }_{ 1 } } { o }_{ 2 }$

$\frac { d }{ { dw }_{ 1 } } { o }_{ 1 }=\frac { d{ o }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { dh }_{ 1 } } \frac { { dh }_{ 1 } }{ { dzh }_{ 1 } } \frac { { dzh }_{ 1 } }{ { dw }_{ 1 } } $

$ \frac { d }{ { dw }_{ 1 } } { o }_{ 1 }={ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 7 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 1 }$

$\frac { d }{ { dw }_{ 1 } } { o }_{ 2 }=\frac { d{ o }_{ 2 } }{ { dzo }_{ 2 } } \frac { { dzo }_{ 2 } }{ { dh }_{ 1 } } \frac { { dh }_{ 1 } }{ { dzh }_{ 1 } } \frac { { dzh }_{ 1 } }{ { dw }_{ 1 } }$

$\frac { d }{ { dw }_{ 1 } } { o }_{ 2 }={ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 8 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 1 }$

$\frac { dE }{ { dw }_{ 1 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 7 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 1 }+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 8 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 1 }$

$\frac { dE }{ { dw }_{ 2 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { d }{ { dw }_{ 2 } } { o }_{ 1 }+({ o }_{ 2 }-{ t }_{ 2 })\frac { d }{ { dw }_{ 2 } } { o }_{ 2 }\\ \frac { d }{ { dw }_{ 2 } } { o }_{ 1 }=\frac { d{ o }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { dh }_{ 2 } } \frac { { dh }_{ 2 } }{ { dzh }_{ 2 } } \frac { { dzh }_{ 2 } }{ { dw }_{ 2 } }$

$\frac { d }{ { dw }_{ 2 } } { o }_{ 1 }={ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 1 }$

$\frac { d }{ { dw }_{ 2 } } { o }_{ 2 }=\frac { d{ o }_{ 2 } }{ { dzo }_{ 2 } } \frac { { dzo }_{ 2 } }{ { dh }_{ 2 } } \frac { { dh }_{ 2 } }{ { dzh }_{ 2 } } \frac { { dzh }_{ 2 } }{ { dw }_{ 2 } }$

$\frac { d }{ { dw }_{ 2 } } { o }_{ 2 }={ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 10 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 1 }$

$\frac { dE }{ { dw }_{ 2 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 1 }+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 10 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 1 }$

$\frac { dE }{ { dw }_{ 3 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 7 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 2 }+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 8 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 2 }$

$\frac { dE }{ { dw }_{ 4 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 2 }+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 10 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 2 }$

$\frac { dE }{ { dw }_{ 5 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 7 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 3 }+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 8 }{ h }_{ 1 }(1-{ h }_{ 1 }){ x }_{ 3 }$

$\frac { dE }{ { dw }_{ 6 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 }){ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 3 }+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 }){ w }_{ 10 }{ h }_{ 2 }(1-{ h }_{ 2 }){ x }_{ 3 }$

$\frac { dE }{ { db }_{ 1 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { db }_{ 1 } } +({ o }_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { db }_{ 1 } }$

$\frac { d{ o }_{ 1 } }{ { db }_{ 1 } } =\frac { { do }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { db }_{ 1 } } ={ o }_{ 1 }(1-{ o }_{ 1 })\frac { { dzo }_{ 1 } }{ { db }_{ 1 } } $

$\frac { { dzo }_{ 1 } }{ { db }_{ 1 } } =\frac { d }{ { db }_{ 1 } } { w }_{ 7 }{ h }_{ 1 }+\frac { d }{ { db }_{ 1 } } { w }_{ 9 }{ h }_{ 2 }={ w }_{ 7 }\frac { d{ h }_{ 1 } }{ { db }_{ 1 } } +{ w }_{ 9 }\frac { d{ h }_{ 2 } }{ { db }_{ 1 } } =$

$ ={ w }_{ 7 }\frac { { dh }_{ 1 } }{ { dzh }_{ 1 } } \frac { { dzh }_{ 1 } }{ { dh }_{ 1 } } +{ w }_{ 9 }\frac { { dh }_{ 2 } }{ { dzh }_{ 2 } } \frac { { dzh }_{ 2 } }{ { db }_{ 1 } } =$

$={ w }_{ 7 }({ h }_{ 1 }(1-{ h }_{ 1 })\cdot 1)+{ w }_{ 9 }({ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 })\cdot 1)$

$\frac { { do }_{ 1 } }{ { db }_{ 1 } } ={ o }_{ 1 }(1-{ o }_{ 1 })({ w }_{ 7 }{ h }_{ 1 }(1-{ h }_{ 1 })+{ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 }))$

$\frac { { do }_{ 2 } }{ { db }_{ 1 } } =\frac { { do }_{ 2 } }{ { dzo }_{ 2 } } \frac { { dzo }_{ 2 } }{ { db }_{ 1 } } ={ o }_{ 2 }(1-{ o }_{ 2 })\frac { { dzo }_{ 2 } }{ { db }_{ 1 } } $</p>

$\frac { { dzo }_{ 2 } }{ { db }_{ 1 } } =\frac { d }{ { db }_{ 1 } } { w }_{ 8 }{ h }_{ 1 }+\frac { d }{ { db }_{ 1 } } { w }_{ 10 }{ h }_{ 2 }={ w }_{ 8 }\frac { d{ h }_{ 1 } }{ { db }_{ 1 } } +{ w }_{ 10 }\frac { { dh }_{ 2 } }{ { db }_{ 1 } } =$

$={ w }_{ 8 }\frac { { dh }_{ 1 } }{ { dzh }_{ 1 } } \frac { { dzh }_{ 1 } }{ { db }_{ 1 } } +{ w }_{ 10 }\frac { { dh }_{ 2 } }{ { dzh }_{ 2 } } \frac { { dzh }_{ 2 } }{ { db }_{ 1 } } =$

$={ w }_{ 8 }({ h }_{ 1 }(1-{ h }_{ 1 })\cdot 1)+{ w }_{ 10 }({ h }_{ 2 }(1-{ h }_{ 2 })\cdot 1)$

$\frac { { do }_{ 2 } }{ { db }_{ 1 } } ={ o }_{ 2 }(1-{ o }_{ 2 })({ w }_{ 8 }{ h }_{ 1 }(1-{ h }_{ 2 })+{ w }_{ 10 }{ h }_{ 2 }(1-{ h }_{ 2 }))$

$\frac { { dE } }{ { db }_{ 1 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 })({ w }_{ 7 }{ h }_{ 1 }(1-{ h }_{ 1 })+{ w }_{ 9 }{ h }_{ 2 }(1-{ h }_{ 2 }))+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 })({ w }_{ 8 }{ h }_{ 1 }(1-{ h }_{ 2 })+{ w }_{ 10 }{ h }_{ 2 }(1-{ h }_{ 2 }))$

$\frac { dE }{ { d }b_{ 2 } } =({ o }_{ 1 }-{ t }_{ 1 })\frac { { do }_{ 1 } }{ { db }_{ 2 } } +(o_{ 2 }-{ t }_{ 2 })\frac { { do }_{ 2 } }{ { db }_{ 2 } } $

$\frac { { do }_{ 1 } }{ { db }_{ 2 } } =\frac { { do }_{ 1 } }{ { dzo }_{ 1 } } \frac { { dzo }_{ 1 } }{ { db }_{ 2 } } ={ o }_{ 1 }(1-{ o }_{ 1 })\cdot 1\\ \frac { { do }_{ 2 } }{ { db }_{ 2 } } =\frac { { do }_{ 2 } }{ { dzo }_{ 2 } } \frac { { dzo }_{ 2 } }{ { db }_{ 2 } } ={ o }_{ 2 }(1-{ o }_{ 2 })\cdot 1$

$ \frac { { dE } }{ { db }_{ 2 } } =({ o }_{ 1 }-{ t }_{ 1 }){ o }_{ 1 }(1-{ o }_{ 1 })+({ o }_{ 2 }-{ t }_{ 2 }){ o }_{ 2 }(1-{ o }_{ 2 })$

In [3]:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

#iterations
epochs = 10000

#learn rate
alpha = 0.01

#weights
w1 = 1
w2 = 1
w3 = 1
w4 = 1
w5 = 1
w6 = 1
w7 = 1
w8 = 1
w9 = 1
w10= 1

#bias
b1 = 1
b2 = 1

#input
x1 = 1
x2 = 20
x3 = 50

#target
t1 = 0.1
t2 = 0.8


# Activation (sigmoid) function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

class Perceptron:
    def __init__(self,X,w,activation = None):
        self.X = X
        self.w = w
        self.activation = activation

    def out(self):
        self.zh()
        self.h = self.activation(self.net)
        return self.h

    def zh(self):
        self.net = np.dot(self.X, self.w)

def forwardProp(x1,x2,x3,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10,b1,b2):

    p1 = Perceptron([x1,x2,x3,1],[w1,w3,w5,b1],sigmoid)
    p2 = Perceptron([x1,x2,x3,1],[w2,w4,w6,b1],sigmoid)
    p3 = Perceptron([p1.out(),p2.out(),1],[w7,w9,b2],sigmoid)
    p4 = Perceptron([p1.out(),p2.out(),1],[w8,w10,b2],sigmoid)

    h1 = p1.out()
    h2 = p2.out()
    o1 = p3.out()
    o2 = p4.out()

    return h1, h2, o1, o2

def gradients(h1, h2, o1, o2):

    #gradients of weights
    dE_dw7  = (o1-t1)*o1*(1-o1)*h1
    dE_dw8  = (o2-t2)*o2*(1-o2)*h1
    dE_dw9  = (o1-t1)*o1*(1-o1)*h2
    dE_dw10 = (o2-t2)*o2*(1-o2)*h2

    dE_dw1 = (o1-t1)*o1*(1-o1)*w7 *h1*(1-h1)*x1 + \
             (o2-t2)*o2*(1-o2)*w8 *h1*(1-h1)*x1
    dE_dw2 = (o1-t1)*o1*(1-o1)*w9 *h2*(1-h2)*x1 + \
             (o2-t2)*o2*(1-o2)*w10*h2*(1-h2)*x1
    dE_dw3 = (o1-t1)*o1*(1-o1)*w7 *h1*(1-h1)*x2 + \
             (o2-t2)*o2*(1-o2)*w8 *h1*(1-h1)*x2
    dE_dw4 = (o1-t1)*o1*(1-o1)*w9 *h2*(1-h2)*x2 + \
             (o2-t2)*o2*(1-o2)*w10*h2*(1-h2)*x2
    dE_dw5 = (o1-t1)*o1*(1-o1)*w7 *h1*(1-h1)*x3 + \
             (o2-t2)*o2*(1-o2)*w8 *h1*(1-h1)*x3
    dE_dw6 = (o1-t1)*o1*(1-o1)*w9 *h2*(1-h2)*x3 + \
             (o2-t2)*o2*(1-o2)*w10*h2*(1-h2)*x3

    dE_db1 = (o1-t1)*o1*( w7*h1*(1-h1) +  w9*h2*(1-h2) ) + \
             (o2-t2)*o2*( w8*h1*(1-h1) + w10*h2*(1-h2) )
    dE_db2 = (o1-t1)*o1*(1-o1) + (o2-t2)*o2*(1-o2)

    if(False):
        print("dE_dw1",dE_dw1)
        print("dE_dw2",dE_dw2)
        print("dE_dw3",dE_dw3)
        print("dE_dw4",dE_dw4)
        print("dE_dw5",dE_dw5)
        print("dE_dw6",dE_dw6)
        print("dE_dw7",dE_dw7)
        print("dE_dw8",dE_dw8)
        print("dE_dw9",dE_dw9)
        print("dE_dw10",dE_dw10)
        print("dE_db1",dE_db1)
        print("dE_db2",dE_db2)

    return dE_dw1, dE_dw2, dE_dw3, dE_dw4, dE_dw5, dE_dw6, dE_dw7, dE_dw8, dE_dw9, dE_dw10, dE_db1, dE_db2 


def error(oList, tList):
        return 0.5 * (np.power(oList[0] - tList[0], 2) + np.power(oList[1] - tList[1], 2))


errList = []

for i in range(epochs):
    h1, h2, o1, o2 = forwardProp(x1,x2,x3,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10,b1,b2)
    dE_dw1, dE_dw2, dE_dw3, dE_dw4, dE_dw5, dE_dw6, dE_dw7, dE_dw8, dE_dw9, dE_dw10, dE_db1, dE_db2 = gradients(h1, h2, o1, o2)

    #compute error
    sse = error([o1, o2], [t1, t2])
    errList.append(sse)

    #update weights
    w1  = w1  - alpha * dE_dw1
    w2  = w2  - alpha * dE_dw2
    w3  = w3  - alpha * dE_dw3
    w4  = w4  - alpha * dE_dw4
    w5  = w5  - alpha * dE_dw5
    w6  = w6  - alpha * dE_dw6
    w7  = w7  - alpha * dE_dw7
    w8  = w8  - alpha * dE_dw8
    w9  = w9  - alpha * dE_dw9
    w10 = w10 - alpha * dE_dw10
    b1  = b1  - alpha * dE_db1
    b2  = b2  - alpha * dE_db2

print("Final result:")
print("Targets:",t1,t2)
print("Outputs:",o1,o2)
print("Final weights:",w1,w2,w3,w4,w5,w6,w7,w8,w9,w10)

pd.DataFrame(errList, columns=['SSE']).plot()
plt.savefig("sse.png")
plt.show()

Final result:
Targets: 0.1 0.8
Outputs: 0.10826251757824877 0.7974962884590129
Final weights: 1.0 1.0 1.0 1.0 1.0 1.0 -0.7120776441249524 1.027600254409115 -0.7120776441249524 1.027600254409115