Cloudbank

Cloudbank is built to be a Computer Vision and AI training platform alternative to its inspiration “OpenAI Universe” where AI agents can be trained to complete task within a video game environment that can run on a linux PC. The Computer Vision component is handled using OpenCV (C++) while it is planned for the Machine Learning component to be handled using Python. Gstreamer (https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html) is used to capture the video game frame and a bash script is used to send simulated keyboard and mouse inputs to the video the game.

What you’ll need.

1. Linux OS (at least ubuntu 14.04)
2. Gstreamer 1.0
3. OpenCV version 3.4.1 (build with tesseract OCR enabled)
4. c++ 11 or greater compiler (Enable Boost library)
5. python 2.7
6. Steam video game client
7. PlayOnLinux with steam client installed (this is used to run Windows only video games on Linux)
8. xdotool

Why opencv C++ and not Python for vision processing ?

The C++ version of OpenCV is much more mature than it’s Python counterpart and the languge is faster in terms of performance and enables you to implement some GPU parallel and multithreading processing into the project.

How is the video game frame captured?

A library named gstreamer is used with the commands contained within a bash script. gstreamer captures the current frame of the game and saves it as ‘current_game_frame.jpg’

Below is a screenshot of gstremer in action:

How and what data is sent across into python?

Data is sent from C++ to Python using a library called ‘Boost.Python’(http://www.boost.org/doc/libs/1_64_0/libs/python/doc/html/tutorial/index.html). What is sent is an identified objects location in the image (x,y coordinates), its feature points and the identified characters/words in the frame and their location points.

For the identified regions of interest the data is in the form of:

[[x coordinate, y coordinate],[BoundBox width, BoundBox height, BoundBox top left hand coorner x coordinate , BoundBox top left hand coorner y coordinate], [feature Points]]
.
.
.

For the identified text in the game the data is in the form of:

[[,x coordinate, y coordinate],[,x coordinate, y coordinate],… ]

How do you send keyboard commands to the video game?

A Python script sends commands to a bash script which in turn uses a tool call xdotool to simulate keyboard commands (http://www.semicomplete.com/projects/xdotool/) and send them to the window containing the title of the video game.

Below is a gif of the image processing and object identification being done on the captured frame, the character is currently being moved randomly via simulated keyboard and mouse inputs being sent to the window from the controller

Evaluate you alogorithm

The current code in CloudBank gives you the option to save and visually evaluate the results of your image processing algorithm within the ‘…/Cloudbank/game_vision/cloudbank_images/‘ file path as well as measure how long it takes the program to processes the image.

Instructions

1. Download and extract the repo
2. Navigate into the downloaded repo and into follow the path “…/game_vision/gstream_command_to_capture_image” file and modyfy the variable “xname” and assign it to the title of the window currently hosting the video game “xname=