Complex Systems Research Laboratory (SoSLab), CMPE, Boğaziçi University

This is a master of science thesis by Doğukan Erenel and advisor Prof. Haluk Bingöl. You can directly contact to Doğukan Erenel for further inquiries about this study.

Acknowledgements

This work is partially supported by Boğaziçi University Research Fund Project BAP-2011-6017.

Abstract

Within the popularity of new interface devices such as accelerometer based game controllers or touch-screen smartphones, the need of new accessibility options for these interfaces have become emergent. Previous studies gave the idea of using accelerometer based gesture recognition system on touch-screen smartphones with accelerometer as a new interface for visually-impaired people to use touchscreen keyboards. However, almost all studies, which have high accuracy results, are used user-dependent classifications or very limited gesture sets. In this study, our aim is to find an alternative approach to classify 20 different gestures captured by iPhone 3GS’s built-in accelerometer and make high accuracy on user-independent classifications. Our method based on Dynamic Time Warping (DTW) with dynamic warping window sizes. The first experimental result, which is obtained from collected data set, gives 96.7% accuracy rate among 20 gestures with 1062 gesture data totally. The second experimental result, which is obtained from 4 visually-impaired people with implemented calculator as end-user test, gives 95.5% accuracy rate among 15 gestures with 720 gesture data totally. Within this work, a design of accelerometer-based recognition system is given as well as its implementation as a gesture recognition based talking calculator and experimental results. See Demo Video : http://vimeo.com/26196932 or http://www.youtube.com/watch?v=VPzg3gHINPs.

Downloads

Data Set (which is explained below) (.plist): Download Data Set
Training results (which is explained below) (.plist): Download Training Result
Gesture Recognition Framework (made with objective-c and XCode) used in both training and classification parts (.framework): Download Framework
Gesture Recognition Project : Download XCode Project
Thesis Document (.pdf) : Thesis in PDF

Data Set

There are 1091 gesture data (grouped according to actual class number) in one XML file with given information;

Age
Profession (0 : No Job)
1. Unknown Job
2. Student
3. Teacher
4. Worker
5. Doctor
6. Engineer
7. Sport Profession
8. Art Profession
9. Other
Education (0 : No Education)
1. Unknown Education
2. Read Write
3. Elementary School
4. High-School
5. UnderGraduate
6. Master Degree
7. Phd Degree
8. Professor
Disability (Y: Yes or N:No)
Sex (M: Male, F: Female, ?)
Hand usage (L: Left, R: Right)
Current hand usage (L: Left, R: Right)
3D Accelerometer data for given gesture type with 4 data sequences respectively;
1. Timestamp for each accelerometer update clock
2. Gesture data sequence in X axis (time series in x axis)
3. Gesture data sequence in Y axis (time series in y axis)
4. Gesture data sequence in Z axis (time series in z axis)
Date of data collected

Collected gestures are given below;

Gestures used in training and calculator are given below;

Experimental results

In this study; two experiments were made for validating our recognition system. In the first experiment; training set were collected from 15 participant, who do not have any disability, with totally 1091 gesture data. However, in the data validation step the system eliminated 29 gesture data, so 1062 gesture data were used totally. After data collection our system was trained and validated with leave-one-out cross validation. For the second experiment, a talking calculator was made, which uses the implemented framework. Here, Flite text-to-speech library and its wrapper by Sam Foster was used. In order to test our system with the calculator; 40 calculations, containing totally 180 characters, were defined. Then 4 visually-impaired people tested the system by making defined calculations using our calculator in 7 days, here the number of days is determined intuitively. In each day, all participants calculation data were collected in order to check the learning curve and the accuracy of the classifications.