Tensorflow-RL

Tensorflow based implementations of A3C,
PGQ,
TRPO,
DQN+CTS,
and CEM
originally based on the A3C implementation from https://github.com/traai/async-deep-rl. I extensively refactored most of the code and beyond the new algorithms added several additional options including the a3c-lstm architecture, a fully-connected architecture to allow training on non-image-based gym environments, and support for continuous action spaces.

The code also includes some experimental ideas I’m toying with and I’m planning on adding the following implementations
in the near future:

• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• FeUdal Networks for Hierarchical Reinforcement Learning*
• Prioritized Experience Replay
• Neural Episodic Control*

*currently in progress

Notes

• You can find a number of my evaluations for the A3C, TRPO, and DQN+CTS algorithms at https://gym.openai.com/users/steveKapturowski. As I’m working on lots of refactoring at the moment it’s possible I could break things. Please open an issue if you discover any bugs.
• I’m in the process of swapping out most of the multiprocessing code in favour of distributed tensorflow which should simplify a lot of the training code and allow to distribute actor-learner processes across multiple machines.
• There’s also an implementation of the A3C+ model from Unifying Count-Based Exploration and Intrinsic Motivation but I’ve been focusing on improvements to the DQN variant so this hasn’t gotten much love

Running the code

First you’ll need to install the cython extensions needed for the hog updates and CTS density model:

./setup.py install build_ext --inplace

To train an a3c agent on Pong run:

python main.py Pong-v0 --alg_type a3c -n 8

To evaluate a trained agent simply add the —test flag:

python main.py Pong-v0 --alg_type a3c -n 1 --test --restore_checkpoint

DQN+CTS after 80M agent steps using 16 actor-learner threads

A3C run on Pong-v0 with default parameters and frameskip sampled uniformly over 3-4

Requirements

• python 2.7
• tensorflow 1.2
• scikit-image
• Cython
• pyaml
• gym