klips/python/neural-network/README.md

Install required dependencies for matplotlib GUI frontend and all pip other packages for this project

```bash
sudo apt install python3-tk
python3.9 -m pip install -r requirements.txt
```

Neural network implementation using Python CLI to dynamically generate a resizable network
 and then run a given number of learning cycles on the provided data set. 
As an example, the IRIS dataset is used to classify flower types using petal measurements.
Input layer perceptron count can be adjusted with `INPUTS` positional parameter.
Hidden layer perceptron count can be adjusted with `PERCEPTRONS` positional parameter.
Output layer perceptron count can be adjusted with `OUTPUTS` positional parameter.
Hidden layers can be added or removed using`--hidden-layers` option setting.
Node bias can be initialized randomly or with provided data.
Perceptron edge weight bias can be initialized randomly or with provided data.
Threshold for perceptron fire can be initialized randomly or with provided data.

Setup instructions and output of `neural-network.py -h`- 
```bash
python3.9 neural-network.py -h


usage: neural-network.py [-h] [--hidden-layers [HIDDEN_LAYERS]] [--cycles [CYCLES]] [--learn-rate [LEARNING_RATE]]
              [--bias [INITIAL_BIAS]] [--weight [INITIAL_EDGE_WEIGHTS]] [--error-threshold [ERROR_THRESHOLD]]
              [--fire-threshold [FIRE_THRESHOLD]] [--spacing [LAYER_SPACING]] [--horizontal] [--silent] [--verbose]
              [--file [file_path]]
              [INPUTS] [PERCEPTRONS] [OUTPUTS]

Neural network implementation

positional arguments:
  INPUTS                Number of inputs for the neural network
                            (default: '3')
                                
  PERCEPTRONS           Number of perceptrons in each hidden layer
                            (default: '8')
                                
  OUTPUTS               Number of outputs for the neural network
                            (default: '3')
                                

optional arguments:
  -h, --help            show this help message and exit
  --hidden-layers [HIDDEN_LAYERS], -l [HIDDEN_LAYERS]
                        Number of hidden layers
                            (default: '1')
                                
  --cycles [CYCLES], -c [CYCLES]
                        Number of cycles to run through the network
                            (default: '3')
                                
  --learn-rate [LEARNING_RATE]
                        Learning rate to use for the network.
                            Must be within range of 0.0 < rate <= 1.0
                            (default: '0.25')
                                
  --bias [INITIAL_BIAS], -b [INITIAL_BIAS]
                        The initial bias to use for perceptrons within the network.
                            Must be within range of -1.0 <= bias <= 1.0
                            If value is unset, bias will be initialized randomly
                                 
  --weight [INITIAL_EDGE_WEIGHTS], -w [INITIAL_EDGE_WEIGHTS]
                        The initial edge weight to use for node connections in the network
                            If value is unset, edge weights will be initialized randomly
                                
  --error-threshold [ERROR_THRESHOLD], --error [ERROR_THRESHOLD]
                        The acceptable error threshold to use for training the network.
                            (default: '0.5')
                                
  --fire-threshold [FIRE_THRESHOLD], --fire [FIRE_THRESHOLD]
                        The fire threshold for perceptrons in the network.
                            If a perceptron's cumulative inputs reach this value, the perceptron fires
                            (default: '0.25')
                            
  --spacing [LAYER_SPACING]
                        Distance between origin of network layers within visualization
                            (default: '2.0')
                                
  --horizontal, --flip  The network visualization will flow left-to-right
                            (default: 'False')
                                
  --silent              Do not show the network visualization, only print output to console
                            (default: 'False')
                                
  --verbose, -v         When this flag is set, error rate and change in weight will be output for each calculation
                            (default: 'False')
                                
  --file [file_path], -f [file_path]
                        Optionally provide a json file to configure any option available through the cli
                            json keys match --long version of each option, where --long-split option key is "long_split" in json
```

Input and output layers are sized by the length of a single input sequence and the number of possible label classifications.
If the length of an input sequence does not match the number of input nodes requested, a warning will show.
If the length of possible label classifications does not match the number of output nodes requested, a warning will show.
In both cases, the program corrects the node count to match the input data / labels, and not the requested node count.

The total number of output labels provided must match the total number of input sequences.

Running NN program uses IRIS data set by default. 
Warnings will be shown if input and output node count is changed without providing new input.
```bash
python3.9 neural-network.py --file input.json --silent


Warning: Input sequences each contain 3 entries but 5 input nodes were requested.
        Using 3 input nodes instead of 5
Warning: Output labels contain 3 possible classifications but 8 output were nodes requested.
        Using 3 output nodes instead of 8
Creating a single layer neural network: 
        Total input nodes: 3
        Number of perceptrons in each hidden layer: 8
        Total output nodes: 3
        Number of hidden layers: 3
        Fire threshold: 0.25
        Error threshold: 0.5
        Learn rate: 0.25
        Initial bias: Random
        Initial edge weights: Random
Network visualization settings: 
        Graph visualization is enabled: False
        Graph visualization is horizontal: True
        Graph visualization is vertical: False
        Graph visualization layer spacing: 2.0
        Test data input count: 150
inputs layer: [0, 1, 2]
hidden layer: [[3, 4, 5, 6, 7, 8, 9, 10], [11, 12, 13, 14, 15, 16, 17, 18], [19, 20, 21, 22, 23, 24, 25, 26]]
outputs layer: [27, 28, 29]
[Cycle 1]       Accuracy: 92.6667%      [139 / 11]
[Cycle 2]       Accuracy: 95.3333%      [286 / 14]
[Cycle 3]       Accuracy: 96.2222%      [433 / 17]
[Cycle 4]       Accuracy: 96.6667%      [580 / 20]
[Cycle 5]       Accuracy: 96.9333%      [727 / 23]
[Cycle 6]       Accuracy: 97.1111%      [874 / 26]
[Cycle 7]       Accuracy: 97.2381%      [1021 / 29]
[Cycle 8]       Accuracy: 97.3333%      [1168 / 32]
[Cycle 9]       Accuracy: 97.4074%      [1315 / 35]
[Cycle 10]      Accuracy: 97.4667%      [1462 / 38]

Correct: 1462    Wrong: 38       Total: 1500
Cycle 1 accuracy: 92.6667%      Cycle 10 accuracy: 97.4667%
4.8% change over 10 cycles      0.48% average change per cycle
```


Running NN program with garbage data in `input-test.json` to test resizing of input / output layers.
A single input sequence is `[0, 1, 0, 1, 1, 1]` which is length of 6, so 6 input nodes are created.
Within the output labels, there are 8 unique labels in the set, so 8 output nodes are created.
The length a single label must match the number of output nodes.
For 8 output nodes, the labels `[1, 0, 0, 0, 0, 0, 0, 0]` and `[0, 1, 0, 0, 0, 0, 0, 0]` are valid.

```bash
python3.9 neural-network.py --file ./input-test.json --silent


Warning: Output labels contain 8 possible classifications but 10 output were nodes requested.
        Using 8 output nodes instead of 10
Creating a single layer neural network: 
        Total input nodes: 6
        Number of perceptrons in each hidden layer: 8
        Total output nodes: 8
        Number of hidden layers: 3
        Fire threshold: 0.25
        Error threshold: 0.5
        Learn rate: 0.25
        Initial bias: Random
        Initial edge weights: Random
Network visualization settings: 
        Graph visualization is enabled: False
        Graph visualization is horizontal: True
        Graph visualization is vertical: False
        Graph visualization layer spacing: 2.0
        Test data input count: 14
inputs layer: [0, 1, 2, 3, 4, 5]
hidden layer: [[6, 7, 8, 9, 10, 11, 12, 13], [14, 15, 16, 17, 18, 19, 20, 21], [22, 23, 24, 25, 26, 27, 28, 29]]
outputs layer: [30, 31, 32, 33, 34, 35, 36, 37]
[Cycle 1]       Accuracy: 35.7143%      [5 / 9]
[Cycle 2]       Accuracy: 39.2857%      [11 / 17]
[Cycle 3]       Accuracy: 40.4762%      [17 / 25]
[Cycle 4]       Accuracy: 41.0714%      [23 / 33]
[Cycle 5]       Accuracy: 41.4286%      [29 / 41]
[Cycle 6]       Accuracy: 41.6667%      [35 / 49]
[Cycle 7]       Accuracy: 41.8367%      [41 / 57]
[Cycle 8]       Accuracy: 41.9643%      [47 / 65]
[Cycle 9]       Accuracy: 42.0635%      [53 / 73]
[Cycle 10]      Accuracy: 42.1429%      [59 / 81]

Correct: 59      Wrong: 81       Total: 140
Cycle 1 accuracy: 35.7143%      Cycle 10 accuracy: 42.1429%
6.4286% change over 10 cycles   0.6429% average change per cycle
```

By default, the following network and visualization will be generated

```bash
python3.9 neural-network.py
# Output removed for GUI example
```
![](screenshot.png)