hardware-rc documentation
=========================

Author: **Andrew Carr**

*As part of a Master's thesis at Cal Poly SLO, 2026*

----

A package for training and analyzing MEMS-based reservoir computers for reinforcement learning tasks.

Specifically, this package is designed for the single-node reservoir computing architecture where reservoirs are sampled
from a single physical system using time-multiplexing. 



----

Overview of package features:
-----------------------------

Algorithms:
   - DQN_RC: Deep Q-Network for discrete action spaces
   - PPO_RC: Proximal Policy Optimization for continuous action spaces

Environments:
   - Compatible with Farama Gymnasium environments
   - Any environment mimicking the Gymnasium API should work

Reservoir Simulation:
   - MEMS dynamics simulated with a custom Runge-Kutta 4th order DDE solver implemented in JAX for speed on CPU.
   - Masks are generated automatically based on environment state dimensions and user-specified hyperparameters.
   - Reservoir DDE can be inputted with correct JIT-compile format (to be added in future updates).

Model Saving:
   - Models are saved as compressed Numpy .npz files containing all necessary information for training and inference, including hyperparameters, model weights, and training metadata.
   - Default save folders are set to "/models/algorithm/env_name-meta[group]"

----


*Single Node Reservoir Computing Architecture for DQN*

.. image::   static/RC_Schematic.jpg
   :width: 80%
   :align: center
   :alt: RC Architecture

*Masked states fed into physical reservoir*

.. image:: static/conf_mask_reservoir_subplots.png
   :width: 50%
   :align: center
   :alt: Configuration Masks and Reservoir Subplots

*Playing environments*

- CartPole-v1 solved by DQN with reservoir rendering:

.. image:: static/CartPole-v1-DQN-Snipe-20260312-173629_reservoir.gif
   :width: 90%
   :align: center
   :alt: CartPole-v1 DQN RC

- MountainCar-v0 and LunarLander-v3 solved with DQN:

.. list-table::
   :widths: 50 50
   :class: borderless

   * - .. image:: static/MountainCar-v0-DQN-Vole-20260325-010536.gif
          :width: 100%
          :alt: MountainCar-v0 DQN RC
     - .. image:: static/LunarLander-v3-DQN-Possum-20260327-011135.gif
          :width: 100%
          :alt: LunarLander-v3 DQN RC

- MountainCarContinuous-v0 and BipedalWalker-v3 solved with PPO:

.. list-table::
   :widths: 50 50
   :class: borderless

   * - .. image:: static/last_Hyena_20260517-004216.gif
          :width: 100%
          :alt: MountainCarContinuous-v0 PPO RC
     - .. image:: static/best_Rattlesnake-20260504-150344.gif
          :width: 100%
          :alt: Rattlesnake BipedalWalker-v3

----



.. toctree::
   :maxdepth: 2
   :caption: Contents:

   installation
   usage
   hyperparameter_overview

.. toctree::
   :maxdepth: 2
   :caption: API Reference:

   api/index
   api/hardware_rc