Turning smartphones into browser controllers.
Smartphones are ubiquitous, incredibly optimized and cost-effective devices. They have more computing power than a Raspberry Pi, are reliably connected to the internet and include a lot of sensors. This projects aims to turn smartphone into smart controllers for interactive experiments. The original idea was to only use video from phone camera but it soon showed that many other kinds of controllers could be developed.
The main tool to make a peer-to-peer connection is Peerjs, WebRTC.
To generate QRcodes with custom Urls Qrcodejs was used.
Other tools to create the demos themselves are Matterjs for physics simulation, Nipplejs joystick and Handpose for handtracking.
The whole process requires only 3 steps:
Navigate to the PC side website. This will create a peer listening for connection and show a QRcode with a unique ID.
Scan the QRcode with your smartphone. Two websites communicate by eastablishing peer to peer connection. The QRcode will provide all necessary details for the phone to connect to the PC.
After opening the QRcode URL you are ready to use your smartphone as a controller. No extra installation required. The smartphone browser will package your input and send it to the PC browser which will process it and turn it into real actions.
Tiny platformer game with NES controller. Collect the gold coins and jump on the enemies to beat the level.
See the source code: game and controller
OR
Try the tiny platformer
3D racing game controlled with phone joystick. Use the joystick to steer, brake and jump. How many points can you score before the time runs out?
See the cource code: game and joystick
OR
Try the 3D Racer
A physics simulator created with Matterjs. Use the touchpad to select and stack blocks.
See the cource code: game and touchpad
OR
Try the physics simulator
Multiplayer tower building. This demo requires two people to connect. Both players will need to cooperate by selecting the same block to be able to carry it.
See the cource code: game and touchpad
OR
Try the cooperative physics simulator
This demo lets you stream a video into browers that will track your hand to control the ball.
See the cource code: game and video stream
OR
Try the hand tracking
I am currently working on developing an open source library that will offer tools to create an easy peer connection and process the incoming data. Another plan is to expand the range of controller types offered.
This project was created during my summer studentship funded by University of Glasgow.
Big thank you to my supervisor, Dr Jonathan Grizou, for providing guidance and a lot of helpful tips and ideas.