So as mentioned we built a project on the Kinect which helped to teach kids emotion through the use of the Kinect sensor. The project was a game called Oh! and consisted of three levels;
- Ask kids to match the picture card of an emotion to the one being shown, “”Can you match the emotion””
- Ask kids to pronounce the emotion, “”This is a happy face can you say happy””
- Ask kids to make the same emotion as shown on the screen and use the Kinect sensor to detect if they made the correct emotion, “”Can you make a happy face””
We managed to code all three levels however the levels were not glued together yet so to speak. But the basic structure of each level had been coded.I was very impressed with level three as this is a hard task to teach Children with Autism and I would love to see a game like this make its way into therapy use.
The other thing that we did was the game was to customise the game. This meant that the therapist or parent could take a photo of themselves and substitute the face on the screen with their own. This would enable the child to see and also hear a familiar person, this is very important in therapy with Autistic children and helps them to connect with the task.
Another aspect was distraction, children with Autism can get easily distracted with the Kinect you can use the motion sensor to detect when the child moves away with with eyes or with body movement from the screen. In order to help pull the child’s interest back into the game we devised a way that a parent or therapist could chose a you tube game which would play on the screen. This would be one of the child’s favorites as with Autistic children they tend to have favorutie topics that they are interested in and these interests can change. So if the child is interested in Mind Craft this month then the parent can select Minecraft clips, next month if it is Sponge Bob then the parent can change the clips to show Sponge Bob.
Another interesting thing we did was connect it up to the patients like me website, http://www.patientslikeme.com/ We captured various statistics like test scores etc to report to the website so parents and therapists can tracks a child’s progress over the length of the game. One interesting area we did was capture the child’s voice from level two where they needed to speak back to the Kinect. This voice could be captured via audio file and by using an emotion sensor on the audio we could detect¬†whether¬†the child was happy or sad or frustration etc.
Overall I think we had a great idea and also as-well thought out idea for a topic that is highly needed, aka teaching emotions. I also thought the connecting for the further reporting was something that no other team was able to capture and report on.
I contributed the presentation and also workflows for the project, I have attached them below;