...that a particular program just tells the CPU to fetch the info from a specific address and the program defines how to treat it.
Exactly. But RAM is not read "sequentially", and it stands for Random Access Memory which is exactly the opposite.
Besides knowing what a byte is, you don't even know if it's a byte, or a fragment of a larger item like a floating-point number.
I'd like to add to other answers by giving some specific examples.
01000001. The program might copy it from one place to another as part of a large parcel of data without any regard to its meaning. But copying that to the address used by the text-mode video buffer will cause the letter
A to show in some position on the screen. The exact same action when the card is in a CGA graphics mode will display a red pixel and a blue pixel.
In a register, it could be the number 65 as an integer. Doing arithmetic to set the 32's bit could mean anything without context, but might specifically be changing a letter to lower case.
The 8086 CPU (still) has special instructions called DAA※ that is used when the register holds 2 decimal digits, so if you just used that instruction you are interpreting it as two digits
Programs crash because a memory word is read thinking it is a pointer when something otherwise was stored there.
Using a debugger, inspecting memory, a map is used to guide the interpretation for display. Without this symbol information, a low-level debugger lets you specify: show this address as 16-bit words, show this address as long floating point, as strings... whatever. Looking at a network packet dump or unknown file format, puzzling it out is a challenge.
That's a major source of power and flexibility in modern computer architecture: a memory cell can mean anything, data or instruction, implicit only in what it "means" to the program by what it does with the value and how that affects subsequent operations. meaning is deeper than integer width: are these characters ... characters in ascii or ebcdic? Forming words in English or SQU product codes? The address to send to or the return address it came from? The lowest level interpretation (logical bits; integer-like, signed or unsigned; float; bcd; pointer) is contextual at the instruction-set level, but you see that it's all context at some level: the to address is what it is because of the location it's printed on the envelope. It is contextual to the rules of the postman, not the CPU. The context is one big continuum, with bits on one end of it.
※ Footnote: The DAA instruction is encoded as a byte
00100111. So that byte is the aforenamed instruction if read in the instruction stream, and the digits
27 if interpreted as bcd digits, and 0x27 = 39 as an integer, which is the numeral 9 in ASCII, and part of the interrupt table (half of INT 13 2-byte address, used for BIOS service routines).