Hyperparameter Overview

Hyperparameters define algorithms and reservoir properties

One dictionary of all hyperparameters is passed into a DQN_RC or PPO_RC class; however, they are broken between reservoir, DQN-specific, PPO-specific, and seeds.

Reservoir hyperparameters

  • N (int): (reservoir size) The number of neurons in the reservoir

  • theta (float): (neuron separation) The non-dimensional time between sampled neurons or the length of one period of the masked input signal

  • amplification (float): The value used to amplify the input signal to the reservoir.

  • h (float): Simulation step size.

  • InputConnectivity (float): The probability that any mask value that will be set to 0. This determines the connectivity of the reservoir.

  • SampleDelay (int): The number of integration steps to wait before measuring the reservoir response to an input.

  • tau (float): (feedback delay) The delay of feedback in number of thetas.

  • fb_gain (float): (feedback strength) The value to scale feedback.

  • NormalizationFactor (list): A list that determines the divider for normalizing states.

  • NormalizationOffset (list): A list that determines the offset for normalizing states.

Shared algorithm hyperparameters

  • trials (int): the number of training iterations.

  • learning_rate (float): the learning rate for readout updates.

  • learning_rate_decay (float): the value multiplying by learning rate each trial slowly decaying learning rate.

  • learning_rate_min (float): the minimum learning rate that the learning rate will decay to.

  • gamma (float): the discount factor for future rewards.

  • val_size (int): the number of seeds to test the model during validation.

DQN-specific hyperparameters

  • epsilon (float): the epsilon value for epsilon-greedy action selection.

  • epsilon_decay (float): the value multiplying by epsilon each trial slowly decaying epsilon.

  • epsilon_min (float): the minimum epsilon that the epsilon will decay to.

  • target_update_freq (int): the number of trials between updates to the target network.

  • buffer_size (int): the maximum size of the replay buffer.

  • batch_size (int): the number of samples per batch for gradient updates.

PPO-specific hyperparameters

  • N_envs (int): the number of parallel environments to run for collecting data.

  • T_horizon (int): the number of steps to run in each environment per trial before updating the policy.

  • num_epochs (int): the number of epochs to train for each trial.

  • minibatch_size (int): the number of samples per minibatch for gradient updates.

  • clip_eps (float): the epsilon value for PPO clipping.

  • v_clip_eps (float): the epsilon value for value function clipping.

  • lamb (float): the lambda value for GAE advantage estimation.

  • value_coef (float): the coefficient for value loss in the total loss calculation.

  • entropy_coef (float): the coefficient for the entropy bonus in the total loss calculation

  • entropy_coef_step (float): the rate of decay for the entropy coefficient.

  • entropy_coef_min (float): the minimum value for the entropy coefficient after decay.

  • max_grad_norm (float): the maximum value for gradient clipping.

  • log_std_min (float): the minimum value for the log standard deviation of the policy’s action distribution.

  • log_std_max (float): the maximum value for the log standard deviation of the policy’s action distribution.

Seeds

  • general_seed (int): The seed that can set all randomness: RC, mask, and environment. Setting this other than 0, will override other specific seeds.

  • rc_seed (int): The seed for reservoir computer dynamics and mask generation.

  • env_seed (int): The seed for environment randomness.

  • mask_seed (int): The seed for reservoir mask generation.

Example hyperparameter dictionaries

DQN RC for CartPole-v1:

hp = {
    'general_seed': 1,
    'rc_seed': 1,
    'mask_seed': 1,
    'weight_seed': 1,
    'env_seed': 1,
    'N': 300,
    'bufferLength': 6000,
    'learning_rate': 1e-4,
    'learning_rate_decay': 1,
    'learning_rate_min': 1e-5,
    'epsilon': 1,
    'epsilon_decay': 0.999,
    'epsilon_min': 0.01,
    'gamma': 0.99,
    'rewardNormalizationFactor': 1,
    'target_update_rate': 4,
    'batch_size': 16,
    'theta': 1,
    'h': 0.02,
    'SampleDelay': 0,
    'InputConnectivity': 0.2,
    'NormalizationFactor': [4.8*2,3+4,2*0.418,2+4],
    'NormalizationOffset': [4.8,4,0.5,4],
    'trials': 300,
    'amplification': 360,
    'tau': 0,
    'fb_gain': 0,
    'tau_N': 0,
    'val_size': 20,
}

PPO RC for MountainCarContinuous-v0:

hp = {
    'rc_seed': 1,
    'env_seed': 1,
    'N_envs': 10,
    'T_horizon': 2048, # was 20
    'gamma': 0.99, # was .97
    'lamb': 0.92,
    'clip_eps': 0.15,
    'v_clip_eps': 0.2,
    'value_coef': 0.5,
    'entropy_coef': 0.008, # was .01
    'entropy_coef_step': 10e5,
    'entropy_coef_min': 0.002,
    'max_grad_norm': 2,
    'num_epochs': 5,
    'minibatch_size': 128,
    'learning_rate': 0.0003,
    'learning_rate_decay': 1,
    'learning_rate_min': 1e-5,
    'log_std_min': -4,
    'log_std_max': 0,
    'N': 300,
    'theta': 1,
    'h': 0.02,
    'SampleDelay': 3,
    'InputConnectivity': 0.2,
    'NormalizationFactor': [1.2*2, .07*2],  # [1]*2,  #  ,
    'NormalizationOffset': [1.2, .07],  # [0]*2,  #  ,
    'amplification': 200, # was 20 but *10 for mask change and *2 for normalization
    'tau': 0,
    'fb_gain': 0,
    'val_size': 5,
    'trials': 400,
}