Team Name
Cloud.io
Timeline
Spring 2021 – Summer 2021
Students
- Sergio Gonzalez
- Julia Nguyen
- Ryan J Laurents
- Austin W Hansen
- Erick Rattana
- Mohit Tamang
Sponsor
Jesse Lee – Raytheon Technologies
Abstract
The Sensorium Cloud is an ongoing project aiming to provide data visualization for environments without requiring “smart” devices.
The board gather environmental data every few seconds such as temperature, humidity, light reflectivity and more. With each unique data collection, the data is run through a local Machine Learning model which has been trained to predict what is happening in the kitchen. Each set of data and corresponding prediction are sent to AWS for storage.
Once the data is sent to AWS, it is then pulled from AWS to the HoloLens 2 Augmented Reality environment. The view in the device will then display the data for the user to view.
Background
The CSE team split into two sub teams to better represent the goals for this semester, Augmented Reality and Machine Learning.
The Augmented Reality team built upon existing work to include the Machine Learning predictions in the HUD. The possibility of adding object detection which would allow for “floating” predictions beside the item in the physical kitchen itself was researched.
The Machine Learning team worked to move the ML services from AWS to a locally made model. If successful, this change would save the project money and increase the speed of prediction.
Project Requirements
- Data from all sensors on each Sensorium will be sent and stored in an AWS database.
- The Sensorium will use a large variety of sensors to gather datapoints from the environment
- Ability for data from the Sensorium’s current environment to be manipulated and displayed in real-time in an Augmented Reality overlay display in the HoloLens.
- Ability to use machine learning to analyze data and predict if specific events are occurring.
System Overview
●AR side used the HoloLens II as a continuation of an older AR implementation
●ML utilized Scikit-learn to perform local machine learning and predicting. Scikit-learn was utilized due to it being a simple and lightweight ML module.
Results
Currently, boards number 8 and 9 are fully functional. The latest Sensorium board data will push to the cloud, and pull to its respective space on the augmented reality interface. Localized ML is performing at a similar or inferior to machine learning on AWS. Localized machine learning is functioning but has a middling accuracy that ranges from fifty to seventy percent. Duration to predict for each set of data was decreased by eighty percent for both local machine learning and AWS-based machine learning.
Future Work
Implementing an object detection system into the project was suggested by our sponsor towards the end, we were able to successfully deploy an open-source onboard detection model (Doughty), but our attempt to merge with the main code base had an inconclusive result. On its own, the model detected objects successfully, but was very slow. However, when merged, the object detection system’s camera module never loaded, it may be that it was overloading the onboard CPU. It would be best to outsource the object detection computing to a desktop PC, to optimize computing resources on the HoloLens.
For machine learning, we had some success that was inferior or on par with ML done on AWS, thus, future work for local Machine Learning is uncertain. It is however the consensus of the ML team that given larger quantities of data and more time to tune and test, that localized machine learning and categorization using the Sensorium is entirely possible.
Project Deliverables
References
Mitchell Doughty, HoloLens2-Machine-Learning, (2021), https://github.com/doughtmw/Hololens2-Machine-lear
