Let's at first find a polygon edge, closest to the external (relative to the polygon) query point q
. Then, if we really need, we will be able to choose one of the edge ends.
The algorithm is based on external bisectors - rays, which originate at each polygon vertex and divide the angle (which is more than 180
grad) into two equal angles. These bisectors never intersect because of the convexity of the polygon. The guiding vector of such a bisector can be calculated as a half-sum of external normal vectors for two adjacent edges. The closest edge (which we are looking for) together with its two bisectors defines a part of plane, where the query point lies.
We will call such a part of plane a cut wedge - because its vertex was cut by the polygon boundary. In general, we can consider cut wedges, consisting of a polyline (part of the polygon boundary) and two bisectors at its ends. We will need to check if the query point lies inside such a cut edge, but fortunately we don't need to traverse the polyline to perform these checks for each edge - we already know that the query point is outside the polygon. So, we can verify that the query point lies inside the cut wedge checking only its two bisectors. If the point is inside, we construct a trial bisector in the middle of the polyline, and check one of "halves".
So, this a binary search, which starts with a pair of complementary cut wedges (for example, with bisectors, originating from $p_0$ and $p_{n/2}$) and ends when the cut wedge, containing the query point, shrinks to a single edge.