I am really confused about it. I think 32 bits = 4 bytes but 32 bit address is only 1 bit.
The question is confusingly worded, but if the question is "why a pointer of 32 bits (4 bytes) points to only one byte of storage", that is just how a particular computer architecture (like x86) is designed.
It's designed with byte-addressable storage, even though the address space itself is larger than 8 bits, and therefore a pointer into that address space requires more than one byte for its storage.
There is no intrinsic link between the size of the pointer and the size of each addressable unit.
On a 64-bit architecture, it takes 8 bytes (64 bits) to store a pointer which points to one byte.
As for why an 8-bit byte tends to be the minimum addressable unit, the most obvious explanation on the x86 architecture is the need for 16, 32, and 64-bit processors in the x86 lineage (which are all considered to be successors of the 8080, which was an 8-bit processor) to be backward compatible with each previous iteration, going back ultimately to the 8080.
But if you were starting from a clean slate, the number of bits in a byte, and the size of an addressable unit, is somewhat a matter of convention, and there have been historically (and probably still are) different conventions than the 8-bit byte and byte-addressable memory.
Also, with 32-bit processors, although the pointer may address only one byte explicitly, the instruction set is such that a pointer can be treated as addressing a 32-bit value starting at the address referred to by the pointer, but continuing also into 3 subsequent addresses.
This way, a 32-bit pointer can implicitly be addressing a value of 32-bit size, even though it explicitly addresses a value of only 8-bit size. What implicit size a pointer is treated as addressing, depends on the instruction to which that pointer is supplied as an operand.
On top of what everyone else has said, computers tend to be byte-addressable because text is an important use case.
Of course, modern CPUs don't actually read a byte from RAM. They read a cache line, which might be 32-128 bytes in size, and then extract bytes or words from that.
Every so often, a new generation of hardware designers comes along and wonders why we don't just load and store words, and then use a second instruction to extract bytes if we need them. In the next revision of the hardware, without fail, the byte addressing instructions always gets added back. Increasing pressure on the instruction window is a bad idea.