This is one of those terms that can mean different things depending upon who you ask and the context in which it is asked.
Usually a processor's word size is defined as the largest size integer which it can operate on with a single arithmetic instruction. For example, 64 bit computer can add or subtract 64 bits with a single instruction. But that same computer may be able to operate on bytes, 16 bit words, and 32 bit longwords.
Don't equate the memory path with the computer's word size. Often data is moved into and out of cache in a fixed size block that is a multiple of the computer's word size. A 64 bit CPU has 8, 8 bit bytes per word but might use a 64 byte cache line and move data into and out of memory in cache block chunks, even if the CPU is only accessing 1 byte of the cache block. OR conversely, the data path to memory could be smaller, as the case with the old 8088 CPU. It used 16 bit CPU registers and did 16 bit arithmetic, but had an 8 bit data bus. It had to do 2 memory transfers to load or store a register.
And then there are fixed definitions for a word size, for example...
DEC had a problem with just saying WORD especially on VAX architecture, mostly because the company had built so many different computers with different natural word sizes (9, 18, 36, 12, 16, 32 and 64 that I can think of) and the VAX was capable of working with a large variety of data sizes. So they decided to define terms like WORD to have more definite meanings.
Byte = 8 bits,
Word = 16 bits,
Longword = 32 bits,
Quadword = 64 bits,
Octaword = 128 bits
This took much of the confusion out of discussions when a processor was capable of handling multiple different data sizes.
edit - I didn't really address this part of the question:
"And you have a data bus that can pick up 4 contiguous memory locations, bytes, at a time. So I suppose if the CPU wanted address 0, then it'd pick up address #0,#1,#2,#3 And if the CPU wanted address 1, then it'd pick up #1,#2#,3#,#4"
IF a 64 bit computer is capable of addressing Longwords on a byte boundary, normally the system would load the cache block containing the address 0 from memory into cache. Often that will be on a multiple word cache size. So in the example I gave with a 64 byte cache line, the hardware would move the first 64 bytes into cache as a chunk. The CPU would then access the bytes from cache as you describe (well sort of, it may have to shift bytes to get them loaded correctly). If the longword you attempted to load spanned 2 cache blocks, the hardware would have to load 2 cache blocks (128 bytes) to access the 4 you were interested in. Or in the example you gave with a 32 bit data bus, it would have to load 64 bits (2 lines) to get the 32 that you want. If possible, it's best to align data on its natural boundary to avoid this and other data manipulations required to deal with unaligned data.