Team Name
Working in Progress
Timeline
Fall 2023 – Spring 2024
Students
- Farhan Sadiq
- Ron Nguyen
- Sazzadul Islam
- Caleb Myatt
- Scott Frazier
Sponsor
Shawn N. Gieser
Abstract
ThereMelo is an innovative educational tool designed to simplify the learning of musical instruments through advanced technology. Utilizing the Ultra Leap Motion 2 controller, Unity, and FMOD, this application enables users to interact with a virtual theremin by simply using hand gestures. Aimed primarily at young learners, ThereMelo not only makes music more accessible but also introduces users to the exciting integration of STEM and the arts. Through intuitive motion detection, ThereMelo offers a unique musical experience that encourages creativity and demonstrates the potential of combining computer vision with the art of music creation.
Background
The theremin is a unique electronic musical instrument invented early in the 20th century by Leon Theremin. Unlike traditional musical instruments, the theremin does not require physical contact to be played. Instead, it operates on the principle of capacitance, and sound is generated by the movement of the performer’s hands in relation to its two metal antennas, one controlling pitch and the other volume. The sound and distinctive playing technique have made it a curious and fascinating instrument, yet it remains notoriously difficult to master due to its touchless nature and the precise control required over hand movements.
Given the complexities associated with learning and accessing traditional musical instruments like the theremin, ThereMelo was conceived to address these challenges through modern technology. By using the Ultra Leap Motion 2 controller, ThereMelo translates the intricate movements required to play the theremin into a digital format that is more accessible and less intimidating for beginners. This approach not only lowers the barrier to entry for engaging with music creation but also introduces a novel way to interact with technology.
Moreover, the educational potential of ThereMelo extends beyond just music. The project is rooted in a broader vision of STEM education, where students are not only consumers of technology but also creators. By integrating elements of computer science, physics, and music, ThereMelo provides a rich, interdisciplinary learning experience. It exemplifies how technology can transform traditional education, making abstract concepts tangible and interactive. This is particularly significant in today’s educational landscape, where engaging students in STEM from an early age is crucial to developing the next generation of innovators.
In creating ThereMelo, our team aims to harness the captivating allure of music to introduce young learners to the wonders of science and technology. The application serves as a bridge, linking music with the world of technology, demonstrating that these fields are not mutually exclusive but rather can be complementary and equally enriching. Through ThereMelo, kids and other users can experience the joy of creating music and the thrill of technology, which are at the heart of both musical expression and scientific curiosity.
Project Requirements
- ThereMelo controls shall be like a theremin and to ensure that our product is like that we will be using a real theremin.
- ThereMelo shall be approved for any age group to use.
- ThereMelo shall use the Ultra Leap Motion 2 Controller.
- ThereMelo shall be an executable stand alone application.
- Any fabrication equipment provided used in the development of the project shall be used in accordance application will be outputting sound data for the user to process.
- ThereMelo shall output sound consistent to that of a theremin from the users input.
System Overview
The system architecture of ThereMelo is built around three core technologies: the Ultra Leap Motion 2 controller, Unity, and FMOD. The Leap Motion controller captures the precise movements of the user’s hands. This data is then processed in Unity, which hosts the virtual environment and the theremin model. Interactions within this environment trigger audio responses that are dynamically handled by FMOD, providing a realistic and immersive sound experience. This integration challenges were primarily in syncing the motion data with sound output accurately, which was addressed through iterative testing and calibration.
Results
Below is a simple demonstration for ThereMelo in its current form.
Future Work
Looking ahead, ThereMelo is poised for several exciting enhancements should our team continue development. Plans include expanding the range of instruments available within the app, incorporating machine learning to adapt the learning experience to individual user progress, and implement more lessons to further teach user more complex concepts.
Project Files
Project Charter (link)
System Requirements Specification (link)
Architectural Design Specification (link)
Detailed Design Specification (link)
Source Code (link)
Poster (link)
References
Chris Chronopoulos. Quadrant: A multichannel, time-of-flight based hand tracking interface for computer music. 2021.
Lucas Silva Figueiredo and Veronica Teichrieb. Kinect vs . color gloves : A music game tool for hand tracking evaluation. XI Brazilian Symposium on Games and Digital Entertainment, 2012.
Lucas Raphael MÃller, Jens Petersen, Amine Yamlahi, Philipp Wise, Tim J. Adler, Alexander Seitel, Karl Friedrich Kowalewski, Beat MÃller, Hannes Kenngott, Felix Nickel, and Lena Maier-Hein.
Robust hand tracking for surgical telestration. International Journal of Computer Assisted Radiology and Surgery, 17, 2022.
Na Pan and Na Pan. Research on music wireless control based on motion tracking sensor and internet of things. IEEE Access, 9, 2021.
AleÅ¡ VysockÃ, Stefan Grushko, Petr OÅ¡Äádal, Tomáš Kot, Ján Babjak, Rudolf JánoÅ¡, Marek Sukop, and Zdenko Bobovský. Analysis of precision and stability of hand tracking with leap motion sensor, 2020.
