The question is already stated in the title.
A full answer to this question depends on the underlying computer architecture, as well as on layers of protection that the operating system may add on top of the basic architecture.
In the Harvard architecture instructions and read/write data are held in physically separate memory stores accessed by separate data pathways (buses). It is physically impossible to load a value from the data memory into the instruction register and it is physically impossible for a program to modify its own instructions. Values in the instruction memory may be copied into data registers and then treated as data, but the processor will know this from context e.g. when it sees an opcode "load the next word into Register A", the processor will assume the value in the next address in the instruction memory is a data value to be copied into Register A.
In the more flexible von Neumann architecture (which is adopted by most modern computers) instructions and data are held in the same memory and the processor does not see any difference between them. The processor assumes that whatever memory address its program counter points to at the start of an instruction cycle holds an instruction, and it will load that value into its instruction register and will try to execute it as if it were an instruction.
The danger with the von Neumann architecture is that the processor may treat data that was not intended to be executed as instructions, with unpredictable results. Or it may treat its own instructions as data and change them - this is known as self-modifying code, and is usually avoided because it is difficult to understand, test and maintain.
The solution to this danger is to re-introduce a distinction between data and instructions into the von Neumann architecture by using a no-execute bit, which can be implemented at a hardware level or in the operating system software.
In current architectures, the CPU does not need to distinguish. Whatever is pointed by the program counter register will be interpreted as an instruction and executed. Everything else can be manipulated as "data". For example, you can have a program modify its own code (in fact, some computer viruses do this, with the aim of avoiding detection).
It doesn't. The CPU executes instructions beginning at the program's start address. After each instruction, the CPU either jumps to the address specified by that instruction or, if it's not a jump, moves on to the next instruction in memory. The start address is known to contain executable code because the program was just loaded there. The correctness of the program's logic ensures that each subsequent memory location from which instructions are retrieved actually contains an instruction. However, an error in the program (or memory corruption) could cause the CPU to start fetching "instructions" from memory that is supposed to be interpreted as data, leading to unpredictable behaviour – usually crashing the program.
The computer doesn't know, just as it has no idea whether the information at a certain memory cell should be interpreted as a character, integer or floating point number.
When executing code, the CPU reads the code byte by byte. Each instruction is composed of an opcode followed by relevant information, such as constants (immediates) and addresses. In principle, however, you could jump in the middle of such a code, so the same byte could be use both as an opcode and as data (though in practice this never ever happens).
Adding to other answers: Modern CPU's can mark memory regions as "do not execute" or "do not modify", and the operating system will instruct them to do so. So if you start a program, the operating system should know which part of memory is instructions, and which is data. It will then mark the data as "do not execute" (because you really don't want the CPU to try to execute data as if it were code) and it will mark the instructions as "do not modify" (because you really don't want the CPU to modify the instructions it's executing).
The CPU will obey this. If it is given an instruction that tells it to run code inside a "do not execute" region, it will refuse to do so, and your program will crash (because that is considered safer than trying to execute random data as if they were code), and if it it given an instruction that tells it to modify data in a "do not modify" region, it will also refuse to do so.
Still, the CPU does NOT know whether the "do not execute" region contained code or not, only that it isn't allowed to execute it.
(Many years ago self modifying code was considered really cool and clever. Today, with gazillions of hackers trying to break into a computer system, that ability is considered Dangerous with a capital D).