hardware-rc documentation

Author: Andrew Carr

As part of a Master’s thesis at Cal Poly SLO, 2026

A package for training and analyzing MEMS-based reservoir computers for reinforcement learning tasks.

Specifically, this package is designed for the single-node reservoir computing architecture where reservoirs are sampled from a single physical system using time-multiplexing.

Overview of package features:

Algorithms:

  • DQN_RC: Deep Q-Network for discrete action spaces

  • PPO_RC: Proximal Policy Optimization for continuous action spaces

Environments:

  • Compatible with Farama Gymnasium environments

  • Any environment mimicking the Gymnasium API should work

Reservoir Simulation:

  • MEMS dynamics simulated with a custom Runge-Kutta 4th order DDE solver implemented in JAX for speed on CPU.

  • Masks are generated automatically based on environment state dimensions and user-specified hyperparameters.

  • Reservoir DDE can be inputted with correct JIT-compile format (to be added in future updates).

Model Saving:

  • Models are saved as compressed Numpy .npz files containing all necessary information for training and inference, including hyperparameters, model weights, and training metadata.

  • Default save folders are set to “/models/algorithm/env_name-meta[group]”

Single Node Reservoir Computing Architecture for DQN

RC Architecture

Masked states fed into physical reservoir

Configuration Masks and Reservoir Subplots

Playing environments

  • CartPole-v1 solved by DQN with reservoir rendering:

CartPole-v1 DQN RC

  • MountainCar-v0 and LunarLander-v3 solved with DQN:

MountainCar-v0 DQN RC LunarLander-v3 DQN RC

  • MountainCarContinuous-v0 and BipedalWalker-v3 solved with PPO:

MountainCarContinuous-v0 PPO RC Rattlesnake BipedalWalker-v3

Contents:

API Reference: