Note: I haven't checked the answer carefully yet and there are missing parts to be written, consider it a first draft.
This answer is meant mainly for people who are not researchers in complexity theory or related fields. If you are a complexity theorist and have read the answer please let me know if you notice any issue or have an idea about to improve the answer.
Where you can find claimed solutions of P vs. NP
- There is The P-versus-NP page which has a list of such claims.
- Articles claiming to resolve the question are regularly posted on arXiv.
Other lists of how not to solve P vs. NP
Lance Fortnow, So You Think You Settled P verus NP, 2009
Scott Aaronson, Eight Signs A Claimed P≠NP Proof Is Wrong, 2010
Polymath page for Deolalikar's paper, where the further readings section has nice list of references about the problem.
How not to approach P vs. NP
Let me discuss "how not to approach P vs. NP" not in the sense of ideas that will not work but in a more general sense.
P vs. NP is a easy to state problem (see also my answer here):
NP = P : For every decision problem with a polynomial time verifier algorithm there is an polynomial time algorithm.
or equivalently
There is a polynomial time algorithm for SAT.
SAT can be replace with any other NP-complete problem.
.
Often people oversimplify and overphilosophize the problem and exaggerate the practical importance of the problem (as stated above). Such statements are often meant to give intuition, but they are not in any way a replacement for the actual mathematical statement of the problem.
Theoretical efficiency is not the same as feasibility in practice.
Let me first with exaggerated practical consequences.
I. It is possible that P=NP but it does not help for any problem in practice!
Say for example that SAT is in P but the fastest algorithm for its running time is
$2^{2^{64}} n^{65536} + 2^{2^{128}}$. This algorithm is of no practical use.
II. It is possible that P$\neq$NP and we can solve NP-complete problems efficiently.
Say for example that SAT is not in P but has an algorithm with running time $n^{\lg^*\lg^* n}$.
To give an input that would make $\lg^* n > 6$ you have to use more electrons that there are thought to be in universe. So the exponent is essentially $2$.
The main point here is that P is an abstract simple model of efficient computation, worst-case complexity is an abstract simple model of estimating the cost of a computation, etc. All of these are abstractions, but no one in practice would consider an algorithm like the one in (I) above as an efficient algorithm really. P is a nice abstract model, it has nice properties, it makes technical issues easy, and it is a useful one. However like all mathematical abstraction it hides details that in practice we may care about. There are various more refined models but the more complicated the model becomes the less nice it would be to argue about.
What people care about in practice is to compute an answer to the problem for instances that they care about using reasonable amount of resources. There are task dependent and should be taken into consideration.
Trying to find better algorithms for practical instances of NP-hard problems is an interesting and worthy endeavor. There are SAT-solver heuristic algorithms that are used in the industry and can solve practical instances of SAT with millions of variables. There is even an International SAT Competition.
(But there are also small concrete instances that all these algorithms fail and fail quite badly, we can actually prove that all state of art modern SAT-solvers take exponential time to solve simple instances like propositional Pigeonhole Principle.)
Keep in mind that correctness and running time of programs cannot be obtained just from running the program on instances. It does not matter how many instance you try, no amount is sufficient. There are infinitely many possible inputs and you have to show correctness and efficiency (i.e. running time is polynomial) of the program for all of them. In short, you need mathematical proof of correctness and efficiency. If you do not know what is a mathematical proof then you should first learn some basic mathematics (read a textbook discrete math/combinatorics/graph theory, these are good topic to learn about what is considered a mathematical proof).
Also be careful about other claims about P vs. NP and the consequence of its answers.
Such claims are often based on similar simplifications.
Complexity theorists do not really care about an answer to P vs. NP!
I exaggerated a bit. Of course we do care about an answer to P vs. NP. But we care about it in a context. P vs. NP is our flagship problem but it is not the ultimate goal. It is an easy to state problem, it involves many fundamental ideas, it is useful for explaining the kind of questions we are interested in to people who are not familiar with the topic.
But we do not seek a one bit Yes/No answer to the question.
We seek a better understanding of the nature of efficient computation. We believe that resolving the question will come with such understanding and that is the real reason we care about it. It is part of a huge body of research. If you want to have a taste of what we do have look at a good complexity theory textbook, e.g. Arora and Barak's "Complexity Theory: A Modern Approach" (draft version).
Let us assume that someone comes with an encrypted completely formal proof of P$\neq$NP and we can verify its correctness to a very high degree of confidence by selecting and decrypting a few bits of the proof (see Zero-Knowledge Proof and PCP theorem). So we can verify the claim with probability of error less than a meteor hitting our house, we are quite sure the proof is correct and P=NP, but we do not know the proof. It will not create much satisfying or exciting for us. The formal proof itself will not also be that satisfying. What we seek is not a formal proof, what we seek is understanding.
In short, from a complexity theorist's perspective
P vs. NP is not a puzzle with a Yes/No answer.
We seek an answer to P vs. NP because
we think it will come a better understanding of the nature of efficient computation.
An answer without a major advancement in our understanding is not very interesting.
There has been too many occasions that non-experts have claimed solutions for P vs. NP,
and those claims typically suffer from issues that they would not have made if they just read a standard textbook on complexity theory.
Common problems P=NP
The claims of P=NP seem to be more common. I think the following is the most common type. Someone has an idea and writes a program and tests it on a few instances and thinks it is polynomial time and correctly solves an NP-complete problem. As I explained above no amount of testing will show P=NP. P=NP needs a mathematical proof, not just a program that seems to solve an NP-complete problem in polynomial time.
These attempts typically suffer from one of the two issues:
I. the algorithm is not really polynomial time.
II. the algorithm does not solve all instances correctly.
Signs that a P$\neq$NP argument is not correct
[to be written]
How to check that your algorithm does not really work
You cannot show that your algorithm works correctly by testing. But you can show it does not work correctly by testing! So here is how you can make sure that your algorithm is not correct if you are willing to do some work.
First, write a program to convert instances of SAT (in the standard CNF format) to the NP-hard problem that you are solving. SAT is one of the most studied NP-hard problems and reductions from other problems to SAT is typically easy. Second, take the examples that the state of art SAT-solvers struggle with (e.g. take the examples from SAT competition) and feed them to your algorithm and see how your algorithm performs. Try known hard instances like propositional Pigeonhole Principle (and don't cheat by hard-coding them as special cases), cryptographic instances (like RSA Factoring Challenges), random k-SAT instances near the threshold, etc.
Similarly you can check that your algorithm is not efficient. E.g. if you think your algorithm's running time is not $10 n^2$ but it is taking days to solve an instance of say size 1000. Fix the polynomial worst-case running-time upper bound that you think your algorithm has. Take the instances and estimate the time your algorithm will take to solve them and check if matches your estimates.
How to check your algorithmic P=NP idea cannot work
If you do these you will be pretty sure that your algorithm does not work (if it works better than the state of the art SAT-solvers then compete in the next competition and lots of people would be interested in studying your algorithm and ideas).
Now you know it does not really work but that is not enough. You want to know why,
is the reason my algorithm does not work a small issue that can be fixed or is there a fundamental reason why it cannot work?
Sometimes the problem with the algorithm is simple and one can identify what was wrong conceptually. The best outcome is that you understand the reason your idea cannot work. Often it is not the case, your idea does not work but you cannot figure out why. In that case keep in mind:
understanding why some idea cannot work can be more difficult that solving P vs. NP!
If you can formalize your idea enough you might be able to prove a limitations of particular ideas (e.g. there are results that say particular formalizations of greedy algorithm cannot solve NP-complete problems). However, it is even more difficult, and you do not have much chance if you have not read a standard complexity theory textbook.
Sometime there is not even a clear conceptual idea why the algorithm should work, i.e. it is based on some not well-understood heuristics. If you do not have a clear conceptual idea of why your algorithm should work then you might not have much chance in understanding why it does not!
Common problems with P$\neq$NP claims
Although most experts think P$\neq$NP is more likely than P=NP, such claims seems to be less common. The reason is that proving lower-bounds seems to be a harder task than designing algorithms (but often proving lower-bounds and upper-bounds are intrinsically related).
Issue 1: the author does not know the definition of P and NP, or even worse does not understand what is a mathematical proof. Because the author lacks basic mathematical training he does not understand when he is told what he is presenting is not a proof (e.g. the steps do not follow from previous ones).
Issue 2: the author confuses "we don't know how" with "mathematical impossibility". For example they make various unjustified assumptions and when asked "why this statement is true?" they reply "how can it be false?". One common one is to assume that any program solving the problem has to go throw particular steps, e.g. it has to compute particular intermediate values, because he cannot think of an alternative way of solving the problem.
[to be completed]
Signs that a P$\neq$NP argument is not correct
[to be written]
How to check your P$\neq$NP idea cannot work
If a claim does not suffer from these basic issues then rejecting it becomes more difficult. On the first level one can find an incorrect step in the argument. The typical response from the author is that I can fix it and this back and forth can go on. Similar to P=NP solutions it is often a very difficult to find a fundamental issue with an idea that can show it cannot work, particularly when the idea itself is informal.
In the best case, if we can formalizes the idea and identify the obstacle that shows the idea cannot work we have proven a new barrier result (this is how attempts to prove P$\neq$NP using circuit lower-bounds lead to the Natural Proofs barrier).