Hyperparameter Overview
====

Hyperparameters define algorithms and reservoir properties

One dictionary of all hyperparameters is passed into a DQN_RC or PPO_RC class; however,
they are broken between reservoir, DQN-specific, PPO-specific, and seeds.



Reservoir hyperparameters
----

- **N** (int): (reservoir size) The number of neurons in the reservoir
- **theta** (float): (neuron separation) The non-dimensional time between sampled neurons or the length of one period of the masked input signal
- **amplification** (float): The value used to amplify the input signal to the reservoir.
- **h** (float): Simulation step size.
- **InputConnectivity** (float): The probability that any mask value that will be set to 0. This determines the connectivity of the reservoir.
- **SampleDelay** (int): The number of integration steps to wait before measuring the reservoir response to an input.
- **tau** (float): (feedback delay) The delay of feedback in number of thetas.
- **fb_gain** (float): (feedback strength) The value to scale feedback.
- **NormalizationFactor** (list): A list that determines the divider for normalizing states.
- **NormalizationOffset** (list): A list that determines the offset for normalizing states.

Shared algorithm hyperparameters
-----
- **trials** (int): the number of training iterations.
- **learning_rate** (float): the learning rate for readout updates.
- **learning_rate_decay** (float): the value multiplying by learning rate each trial slowly decaying learning rate.
- **learning_rate_min** (float): the minimum learning rate that the learning rate will decay to.
- **gamma** (float): the discount factor for future rewards.
- **val_size** (int): the number of seeds to test the model during validation.

DQN-specific hyperparameters
----
- **epsilon** (float): the epsilon value for epsilon-greedy action selection.
- **epsilon_decay** (float): the value multiplying by epsilon each trial slowly decaying epsilon.
- **epsilon_min** (float): the minimum epsilon that the epsilon will decay to.
- **target_update_freq** (int): the number of trials between updates to the target network.
- **buffer_size** (int): the maximum size of the replay buffer.
- **batch_size** (int): the number of samples per batch for gradient updates.


PPO-specific hyperparameters
----
- **N_envs** (int): the number of parallel environments to run for collecting data.
- **T_horizon** (int): the number of steps to run in each environment per trial before updating the policy.
- **num_epochs** (int): the number of epochs to train for each trial.
- **minibatch_size** (int): the number of samples per minibatch for gradient updates.
- **clip_eps** (float): the epsilon value for PPO clipping.
- **v_clip_eps** (float): the epsilon value for value function clipping.
- **lamb** (float): the lambda value for GAE advantage estimation.
- **value_coef** (float): the coefficient for value loss in the total loss calculation.
- **entropy_coef** (float): the coefficient for the entropy bonus in the total loss calculation
- **entropy_coef_step** (float): the rate of decay for the entropy coefficient.
- **entropy_coef_min** (float): the minimum value for the entropy coefficient after decay.
- **max_grad_norm** (float): the maximum value for gradient clipping.
- **log_std_min** (float): the minimum value for the log standard deviation of the policy's action distribution.
- **log_std_max** (float): the maximum value for the log standard deviation of the policy's action distribution.


Seeds
----
- **general_seed** (int): The seed that can set all randomness: RC, mask, and environment. Setting this other than 0, will override other specific seeds.
- **rc_seed** (int): The seed for reservoir computer dynamics and mask generation.
- **env_seed** (int): The seed for environment randomness.
- **mask_seed** (int): The seed for reservoir mask generation.


Example hyperparameter dictionaries
----

**DQN RC for CartPole-v1:**

.. code-block:: python

    hp = {
        'general_seed': 1,
        'rc_seed': 1,
        'mask_seed': 1,
        'weight_seed': 1,
        'env_seed': 1,
        'N': 300,
        'bufferLength': 6000,
        'learning_rate': 1e-4,
        'learning_rate_decay': 1,
        'learning_rate_min': 1e-5,
        'epsilon': 1,
        'epsilon_decay': 0.999,
        'epsilon_min': 0.01,
        'gamma': 0.99,
        'rewardNormalizationFactor': 1,
        'target_update_rate': 4,
        'batch_size': 16,
        'theta': 1,
        'h': 0.02,
        'SampleDelay': 0,
        'InputConnectivity': 0.2,
        'NormalizationFactor': [4.8*2,3+4,2*0.418,2+4],
        'NormalizationOffset': [4.8,4,0.5,4],
        'trials': 300,
        'amplification': 360,
        'tau': 0,
        'fb_gain': 0,
        'tau_N': 0,
        'val_size': 20,
    }

**PPO RC for MountainCarContinuous-v0:**

.. code-block:: python

    hp = {
        'rc_seed': 1,
        'env_seed': 1,
        'N_envs': 10,
        'T_horizon': 2048, # was 20
        'gamma': 0.99, # was .97
        'lamb': 0.92,
        'clip_eps': 0.15,
        'v_clip_eps': 0.2,
        'value_coef': 0.5,
        'entropy_coef': 0.008, # was .01
        'entropy_coef_step': 10e5,
        'entropy_coef_min': 0.002,
        'max_grad_norm': 2,
        'num_epochs': 5,
        'minibatch_size': 128,
        'learning_rate': 0.0003,
        'learning_rate_decay': 1,
        'learning_rate_min': 1e-5,
        'log_std_min': -4,
        'log_std_max': 0,
        'N': 300,
        'theta': 1,
        'h': 0.02,
        'SampleDelay': 3,
        'InputConnectivity': 0.2,
        'NormalizationFactor': [1.2*2, .07*2],  # [1]*2,  #  ,
        'NormalizationOffset': [1.2, .07],  # [0]*2,  #  ,
        'amplification': 200, # was 20 but *10 for mask change and *2 for normalization
        'tau': 0,
        'fb_gain': 0,
        'val_size': 5,
        'trials': 400,
    }