Rock, Paper, Scissors Robotic Arm
A robotic hand that plays rock, paper, scissors using AI gesture recognition
Author: Emilia Petre
Github Project Link: https://github.com/UPB-PMRust-Students/fils-project-2026-emilia1881
Description
The purpose of this project is to build an interactive robotic arm that plays Rock, Paper, Scissors using a STM32 Nucleo microcontroller and the Rust programming language. It utilizes a Python-based MediaPipe AI model for real-time gesture recognition via the laptop webcam and communicates the results to the hardware via UART. The robotic hand is 3D-printed and uses servo motors to physically mimic the 3 possibilities - rock, paper or scissors. An OLED display tracks the score. The system integrates an ESP module to enable remote scoreboard access via WiFi. The ESP receives the gesture via WiFi and also decides the hand's gesture.
Motivation
This project combines Computer Vision and Embedded Rust, transforming digital gesture recognition into physical robotic motion. By implementing the classic Rock, Paper, Scissors game, the project creates an interactive experience that demonstrates the power of asynchronous firmware in handling complex, real-time user interactions. So, the motivation was to have a tangible, end-to-end system, that feels like a challenge for both myself in building it and for the people interacting with it afterwards, when they play the game.
Architecture
Log
Weeks 5-6
- Researched possible project ideas and chose the Rock, Paper, Scissors Game
- Researched different development boards and selected the STM32 Nucleo-U545RE-Q as the main microcontroller
- Ordered STM32
Weeks 7-8
- Researched additional hardware components and implementation methods
- Ordered the necessary materials and performed initial functionality tests on the purchased modules
Weeks 9-10
- Searched for a suitable 3D-printable robotic hand that can be adapted to project requirements and looking to start the 3D printing process
- Researched how to connect the hardware components together
Week 11
- Completed the 3D printing process for the robotic hand
- Started integrating the electronic components with the printed structure
Week 12
- I tested the ESP module and the movement of the servos. I encountered some difficulties in making the servos move the fingers properly.
- Initially I planned to use an ESP01 module to simply display the score remotely over WiFi. It was not working great, the connection was sometimes flawed.
Week 13
- I found an ESP32 at home randomly, so I decided to try that. At first I connected it to mu phone's hotspot. Because I don't have great mobile data, I switched to Access Point Mode and created a new WiFi Network, RoboHand.
- I built the HTTP server, the player history, the score display.
- I created the connection between the Python script and the ESP32 code. I made the ESP32 receive the gesture and decide what to play (randomly).
- The first option was to create a game in which the hand memorises the last moves and based on the previous choices of gesture, it computed the next propobaility. I decided it was too annoying for the user.
Week 14
- Finalised the connection between ESP32 and STM32.
- Struggled to make all the fingers work properly moslty.
Hardware
The hardware platform of the project is centered around the STM32 Nucleo-U545RE-Q development board, which acts as the main controller of the robotic arm. The SG90 servo motors are used to move the fingers of the hand and reproduce the rock, paper, and scissors gestures. A 0.96-inch SSD1306 OLED display connected through I2C is used to show the game status (Win/Lose/Draw). Wireless connectivity is provided by an ESP32 (more like a last minute change) WiFi module communicating with the STM32 through UART. Gesture recognition is performed on a laptop using the integrated camera. For prototyping and interconnections, a breadboard and jumper wires are used. The robotic hand itself is manufactured using 3D printing, while the system is powered by a Lipo battery.
Schematics
Bill of materials
| Device | Usage | Price |
|---|---|---|
| STM32 Nucleo-U545RE-Q | Main microcontroller of the system | ~127 RON |
| SG90 Servo Motor | Controls the fingers of the robotic hand | ~ 4 x 10 RON |
| OLED Display 0.96" SSD1306 | Displays score and game status | ~ 17 RON |
| ESP32-WROOM | Wireless communication and remote scoreboard access | owned |
| Breadboard (760 points) | Rapid prototyping and circuit connections | ~ 10 RON |
| Jumper Wires Set | Electrical interconnections between modules | owned |
| LiPo Gens Ace 3S 11.1v 450mAh 30C | Powers servos and electronic modules | ~40 RON |
| Step-Down Voltage Converter 5V 5A | Electrical interconnections between modules | ~ 20 RON |
| 3D Printed Robotic Hand Parts | Mechanical structure of the robotic hand | - |
Software
| Library | Description | Usage |
|---|---|---|
| embassy-stm32 | Hardware abstraction layer for STM32 microcontrollers | Used to control GPIO, PWM, UART, I2C, timers and other peripherals |
| embassy-executor | Async task executor for embedded systems | Runs concurrent firmware tasks such as motor control and communication |
| embassy-time | Time management utilities | Handles delays, timers, and periodic scheduling |
| embassy-sync | Synchronization primitives | Enables safe communication between async tasks |
| cortex-m | Low-level ARM Cortex-M support crate | Used for processor-specific features and interrupts |
| cortex-m-rt | Runtime support for Cortex-M targets | Handles startup code and interrupt vectors |
| embassy-futures | Async utility crate | Used for joining and selecting multiple async tasks |
| ssd1306 | OLED display driver | Controls the SSD1306 scoreboard display over I2C |
| embedded-graphics | 2D graphics library for embedded displays | Draws text, icons, and score information on OLED |
| panic-probe | Panic handler for embedded debugging | Reports firmware crashes during development |
| defmt | Lightweight embedded logging framework | Used for debug messages and diagnostics |
| defmt-rtt | RTT transport for defmt logs | Sends debug output to the host PC |
| mediapipe | Real-time gesture recognition framework | Detects hand landmarks and classifies gestures |
| opencv-python | Computer vision library | Captures webcam frames and preprocesses images |