This is a master of science thesis by Doğukan Erenel and advisor Prof. Haluk Bingöl. You can directly contact to Doğukan Erenel for further inquiries about this study.


This work is partially supported by Boğaziçi University Research Fund Project BAP-2011-6017.


Within the popularity of new interface devices such as accelerometer based game controllers or touch-screen smartphones, the need of new accessibility options for these interfaces have become emergent. Previous studies gave the idea of using accelerometer based gesture recognition system on touch-screen smartphones with accelerometer as a new interface for visually-impaired people to use touchscreen keyboards. However, almost all studies, which have high accuracy results, are used user-dependent classifications or very limited gesture sets. In this study, our aim is to find an alternative approach to classify 20 different gestures captured by iPhone 3GS’s built-in accelerometer and make high accuracy on user-independent classifications. Our method based on Dynamic Time Warping (DTW) with dynamic warping window sizes. The first experimental result, which is obtained from collected data set, gives 96.7% accuracy rate among 20 gestures with 1062 gesture data totally. The second experimental result, which is obtained from 4 visually-impaired people with implemented calculator as end-user test, gives 95.5% accuracy rate among 15 gestures with 720 gesture data totally. Within this work, a design of accelerometer-based recognition system is given as well as its implementation as a gesture recognition based talking calculator and experimental results. See Demo Video : or


Data Set

There are 1091 gesture data (grouped according to actual class number) in one XML file with given information;
  1. Age
  2. Profession (0 : No Job)
    1. Unknown Job
    2. Student
    3. Teacher
    4. Worker
    5. Doctor
    6. Engineer
    7. Sport Profession
    8. Art Profession
    9. Other
  3. Education (0 : No Education)
    1. Unknown Education
    2. Read Write
    3. Elementary School
    4. High-School
    5. UnderGraduate
    6. Master Degree
    7. Phd Degree
    8. Professor
  4. Disability (Y: Yes or N:No)
  5. Sex (M: Male, F: Female, ?)
  6. Hand usage (L: Left, R: Right)
  7. Current hand usage (L: Left, R: Right)
  8. 3D Accelerometer data for given gesture type with 4 data sequences respectively;
    1. Timestamp for each accelerometer update clock
    2. Gesture data sequence in X axis (time series in x axis)
    3. Gesture data sequence in Y axis (time series in y axis)
    4. Gesture data sequence in Z axis (time series in z axis)
  9. Date of data collected

Collected gestures are given below;

Gestures used in training and calculator are given below;

Experimental results

In this study; two experiments were made for validating our recognition system. In the first experiment; training set were collected from 15 participant, who do not have any disability, with totally 1091 gesture data. However, in the data validation step the system eliminated 29 gesture data, so 1062 gesture data were used totally. After data collection our system was trained and validated with leave-one-out cross validation. For the second experiment, a talking calculator was made, which uses the implemented framework. Here, Flite text-to-speech library and its wrapper by Sam Foster was used. In order to test our system with the calculator; 40 calculations, containing totally 180 characters, were defined. Then 4 visually-impaired people tested the system by making defined calculations using our calculator in 7 days, here the number of days is determined intuitively. In each day, all participants calculation data were collected in order to check the learning curve and the accuracy of the classifications.

Confusion matrix generated by first experiment's result

Second experiment and result

Daily Average Accuracy

Daily Average Accuracy Per Person

Daily Average Accuracy Per Symbol

Training Result

Effects of Procedures on Recognition Accuracy (Graphical)

Effects of Procedures on Recognition Accuracy (Tabular)

DTW Distance Threshold Values of each Gesture Type

Warping Window Sizes on X Axis

Warping Window Sizes on Y Axis

Warping Window Sizes on Z Axis

Accelerometer Simulation Videos

Triangle Motion (dynamic camera) - Accelerometer Simulation :

Circle Motion (static camera) - Accelerometer Simulation :

Triangle Motion (static camera) - Accelerometer Simulation :

Circle Motion (dynamic camera) - Accelerometer Simulation :