Automation of Wii Play Target Identification Through OpenCV Using Robot Arm

Team Name

Ganbare

Timeline

Spring 2023 – Summer 2023

Students

• Justin Geisen
• Lam Do
• Jack Gervasi
• Duy Nguyen
• Alexander Olmedo

Abstract

The product will consist of a Mitsubishi robot arm playing a point and shoot game that requires it to identify a target and engage the target. The product will make use of machine learning using OpenCV to identify the target. The robot arm will be able to move towards the target and engage it after the target is identified. This product will be able to keep the audience engaged by allowing an audience member to play against the robot arm in a multiplayer setting. The purpose of the product is to encourage curiosity in engineering majors at UTA in an engaging way.

Background

America is losing its competitive advantage in the manufacturing industry. There is a deficiency of skilled STEM workers that can compete in the global economy. There is also a problem in inspiring American high school students to pursue high-tech careers. More college stem majors could potentially increase the amount of manufacturing accomplished once the engineering majors graduate. In order to create more interest in jobs related to science, technology, engineering, and mathematics, we will use a UR5 robot arm that will be capable of playing a Wii game. The goal is to provide a project to successfully engage American high school students to potentially work in high-tech jobs.

Project Requirements

1. Robot arm is capable of moving
2. Subnet communication
3. Functional projection algorithm
4. Robot physical placement relative to IR sensor
5. Raspberry Pi wiring to Wii remote works properly
6. Raspberry Pi GPIO control and SSH control work properly
7. 3d print of Wii remote is functional
8. Setting up computer with discrete GPU
9. Installation of Python3, OpenCV, ROS2, and Nvidia drivers
10. Computer vision function for identifying targets

System Overview

The system will utilize a robot arm to play a point and shoot game on its own.This system will be engaging to the audience and create curiosity for fields in science, technology, engineering, and mathematics. The solution to the problem entails various scopes: machine learning and the overall programming of the robotic arm. While these scopes may seem difficult, the team believes that they are worthwhile pursuits concerning the project. The pursuit of difficulties will build strength that will be useful in future endeavors, both in career and in life. For the programming of the arm, we will be using Robot Operating System (ROS). ROS is open-source software used in the development of robotic software. Contrary to its name, it is not an operating system, but rather it is a set of software frameworks for robot software development. Even though reactivity is important in robotics development, ROS is also not a real-time operating system. ROS will be scripted on a terminal computer issuing the track and engage commands. The velocity and rotation of the arm will be computed and issued by the terminal computer after receiving validation from the machine learning system. The data that will go to the machine learning system will come from the HDMI splitter. The machine learning system will either be a microcontroller or a computer. The first input of the HDMI splitter will go into the flatscreen television, while the other two inputs will go into the terminal computer and the WiiU itself. Machine learning will be implemented using OpenCV due to its vast libraries and the nature of open-source. Machine learning is a subset of OpenCV that provides easy-to-access classes. When it comes to training, OpenCV has the ability to take input images to learn what the target image is and output a certain confidence interval. A majority of algorithms require all or a high amount of training samples to be valid, but OpenCV’s algorithms can tackle this issue. The terminal computer will then translate the positions of the targets into movements and rotations of the robot arm to aim the Wii U remote. Latency may be an issue, however, but that will have to be addressed in further development. The robot arm did however have a quick velocity due to a cursory glance at the datasheet.

Results

Future Work

Project Files

Project Charter (link)

System Requirements Specification (link)

Architectural Design Specification (link)

Detailed Design Specification (link)

Poster (link)

NOTE: The team shifted from using the Mitsubishi Arm playing Duck Hunt to using the UR5 Arm playing Wii Play Shooting Range.