To understand how to evaluate the accuracy of your algorithm, the best way will be to understand how the result of your algorithm will be used.
It is likely that your algorithm is just a component in a system. After all, it's not like being able to visualize the right-hand picture is somehow a useful and important goal in its own right. Rather, it is likely a useful stepping stone along the way to some more meaningful goal.
For instance, maybe you want to do biometric authentication of the individual, and you are planning to use the palm print for authentication, so it is helpful to first extract the three principle lines of the palm. In that case, you will probably take the result of your transformation and then feed it in as input to the next step of processing; ultimately, the final answer of the authentication process will be "yes, accept this is the right person" or "no, this is not the right person". So, in this example, the ultimate goal is authentication. Therefore, I would suggest that the way to evaluate your algorithm is to evaluate how well the palmprint authentication process works, when it uses your algorithm. We could ask about the false accept rate (if an imposter tries to authenticate, what is the probability he/she is wrongly accepted?) and the false reject rate (if the legitimate individual tries to authenticate, what is the probability he/she is wrongly rejected?). Those are two useful metrics for evaluating the accuracy of biometric authentication, so you could evaluate how accurate the process is when using your particular transformation as one of the stages.
In general, a useful way to evaluate accuracy is to look at how the output of your transformation will be used, find some metric that is meaningful and relevant to that particular application, and then measure that metric when we use the output of your transformation as input to the next stage of processing. Of course, this does mean that we cannot evaluate your transformation in isolation -- but you know what? In some sense, that might be unavoidable.
There is another option, but it will require more work, and may yield results that are harder to interpret. You could hand-label each of the input images with a manually drawn line for each of the three principle lines. Call this the "golden reference" lines.
Then, you could manually or automatically process the output of your transformation to get the three lines (by extracting the center of each of the three broad brush strokes we can see in your right-hand image). Call these the "algorithmically inferred" lines.
Next, you could define some metric for the distance between the golden reference lines and the algorithmically inferred lines: for instance, maybe the average distance, or the root-mean-square distance. Finally, you could evaluate your algorithm on a reasonable test set, using this metric to quantify how close your lines are to the golden reference lines.
The problem with this approach is that it is harder to make sense of the results of such an accuracy metric. How do you tell when your algorithm is "good enough"? Beats me. The only way to know that is to look at how the output of your algorithm will be used -- which gets us back to my original suggestion of evaluating your algorithm as part of a larger system.