# HandyAR finger tip detection and tracking

I am reading this paper Handy AR.

As suggested I am adding certain quotes from paper to get basic idea of what is it about :

1. Intro

In this paper, we present and evaluate a method to use a user’s bare outstretched hand in the same way a cardboard AR marker would be used, enabling spontaneous tangible user interfaces in mobile settings

Steps:

Given a captured frame, every pixel is categorized to be either a skin-color pixel or a non-skin-color pixel. An adaptive skin color-based method 17 is used to segment the hand region. According to the generalized statistical skin color model 14, each pixel is determined to be in the hand region if the skin color likelihood is larger than a constant threshold.
In order to adapt the skin color model to the illumination change, a color histogram of the hand region is learned for each frame and accumulated with the ones from the previous n frames (n = 5 works well in practice). Then the probability of skin color is computed by combining the general skin color model and the adaptively learned histogram.

2. Accurate Fingertip detection

Fingertips are detected from the contour of a hand using a curvature-based algorithm similar to the one described in [2]. We then fit the curvature points to ellipses in order to increase the accuracy. The curvature of a contour point is measured on multiple scale levels in order to detect fingertips with various sizes as follows: The points with higher curvature values than a threshold (on any scale level) are selected as candidates for fingertips.
Since the detection algorithm may produce false positives of fingertips for initial detection, we choose the most frequently detected points above the center of the hand for a certain number of consecutive frames as our final fingertips. Thus, for initial detection, the hand has to be held fingers up, which is the most convenient and by far the most common pose anyway. After fingertips have been detected, we eliminate false positives by tracking a successful configuration over time.

3. FingerTip Tracking

Once fingertips are detected, we track them based on matching the newly detected fingertips to the previously tracked fingertips. Similar to 21, we track the fingertip trajectory by a matching algorithm that minimizes the displacement of pairs of fingertips over two frames. In addition, we use our knowledge about the centroid of the hand blob to effectively handle large movements of the hand as follows: The matching cost is minimized as the formula I mentioned below.

I have a couple of doubts in this :
1. In the 2nd paragraph of the 2nd points of steps:

After fingertips have been detected, we eliminate false positives by tracking a successful configuration over time.

What does 'successful configuration over time' mean? If we already are tracking a successful configuration , then why are there false positives.

2.In the next section, Fingertrip tracking has been explained. From what I understood , it has two parts

• Using a matching algorithm
• Using centroid to effectively handle large movenments.

My doubt is in the second part, I am not able to understand properly how does it handle large movenments and also the correctness proof of that statement.

Formula for fi+1

where fi fi+1 are the sets of N fingertips at the ith and i + 1th frame and
Ci , Ci+1 are centroids at ith and i + th frame.

Citation -

I got this from 1 paper published by Taehee Lee, Tobias Hollerer in 2007 11th IEEE International Symposium on Wearable Computers